Work Reflection 2020-03 to 2021-05

DS

• It is ok if an algorithm does poorly in some cases while doing well in some other cases.
• Null data handling. The importance is to find out what those cases are.
  • Join-safety
    What happen if after a join, a dataframe has 0 rows, Or it has < rows than expected

  • Filter-safety
    What happen if after a filter, a dataframe has 0 rows,

  • Data verification
    Measure upstream data quality
    Verify counts end-to-end (per customer, per user, per incident…)
  • Need to create metrics/dashboards/reports for the data going through
    So we can understand the product/user behavior better
  • Always have a peer to review your analysis before presenting to the team
  • business rules can be if else, or decision tree
  • (need to improve) get familiar with Runtime, Storage, compute, memory complexity

Software engineering

• save work incrementally (both coding and research)
  • Lots more information.
    Things to log:
    • start/end of each step
    • Data size start/end of each step
    • A verbose flag to print out partial outputs
      (it helps debugging a lot)

  • Think and test how your algorithm scales with data. Have a plan for what to do when input data increases.

  • have same (table) names across different applications
  • really need good documentations how an app processes the data so someone else don’t have to go through the code to figure it out
    (think I am doing ok on this part)

  • don’t be afraid to ask questions
    (sometimes I spent hours on something that I think colleague would have figured that out under 15mins)

  • save some scripts if you think you/your team may need to use those in the future

Project Management:

• have an understanding of current project’s business value, timeline, stability
• ask what is the significance if each story, task
• periodically think about how to improve the system
• every sprint/month/quarter, take responsibility thinking what you need to do for the next Sprint/month/quarter
• have a central place to keep track of issues
• demo style
  - Problem statement - Work done - Results - Things to do
• (need to improve).[Agile] sometimes need to adjust the scope of the story
• (need to improve).reach out to external team as early as possible

Others

• (need to improve).how to resolve team’s different opinions