Remove records with NULL values

Remove records with NULL values

  1. Click on the Transform icon, choose Custom transform.

    Glue Jobs

  2. In the Transform – Custom code node, specify the following information:

    • Node properties tab, Name - Remove Records with NULL

    • Add the following in the Code block:

      def MyTransform (glueContext, dfc) -> DynamicFrameCollection:
   df = dfc.select(list(dfc.keys())[0]).toDF().na.drop()
   results = DynamicFrame.fromDF(df, glueContext, "results")
   return DynamicFrameCollection({"results": results}, glueContext)

    Glue Jobs

  3. Click on the Transform icon, choose SelectFromCollection.

    Glue Jobs

  4. Specify the following information:

    • Node properties tab, Name – SelectFromCollection
    • Transform tab, Frame index – 0

    Glue Jobs

  5. Remember to Save your work.