I. Information on AIS Data Week 9 – 13 March 2020 (recording and PPTs): 

https://unstats.un.org/unsd/trade/events/2020/AisDataWeek/default.asp

II. Information on Introductory AIS Training Session (2019):

The aim of the online training session is to enable the participants to run the Python code to estimate the indicators Traffic in Port and Time Spend in Port for the UK case.

Date: 17 September 2019 at 7-10.35 AM EST

Optional:
- Basic Knowledge of Python and/or Spark
- Access to AIS dataset in UNGP

Agenda and Materials:

Training Video: https://www.youtube.com/playlist?list=PLnhLUm82UgvBks3_uF3j3_z8PDrY0zIoQ (Playlist with 6 parts)

Training Chat Transcript


Time

Module

Description

Presenter

Session I (1/4 hour)

Overview of AIS Dataset

•What is AIS?

•Description of AIS data variables

Ann-Kristin (UNSD)



Session II (1/4 hour)

Short introduction to the usage of AIS on the UN Global Platform

•How to log in

•Where to find the data

•How to find a script

Session III (1/4 hour)

Introduction to GitLab of AIS Task Team

•What is GitLab?

•Structure of git folder

•How to use

Sean (UNSD)

Session IV (1/2 hour)

Data Preparation and Cleaning

•Loading the data

•Extracting desired variables

•Grouping of time periods

•Checking data, especially gaps in the data

•Cleaning the data until no or only a small number of gaps are identified

Alex (UK ONS)

UKCode.zip



Session V (1 hour)

Calculation of Port Traffic and Time-Spent in Port (based on UK Ports)

•Description of variables

•Application of appropriate filters

•Merge of information from other data sources

•Excursus: Introduction to Spark

Session VI (1/4 hour)

Adapting the codes for other ports

•Discuss the parts that need to change to apply the code for other countries

Session VII (1/4 hour)

Future work and closing remark

•AIS handbook

•Linking Ship ID to Ship Register
•Estimation of Payload
•Mashing with trade data

Markie



  • No labels

1 Comment

  1. UNSD_MM

    ##UPDATE TO AIS UNGP CONNECTION OCT 2019##


    Scala:

     

    Change:


    import com.vividsolutions.jts.geom._ to import org.locationtech.jts.geom._


    Update HBase connection parameters to:


    val feature = "orbcomm"

    val catalog = "ons-historical"

    val hbase = "hbase.optix-ons-local:2181"


    val dataFrame = spark.read.format("geomesa")

        .option("hbase.zookeepers", hbase)

        .option("hbase.catalog", catalog)

        .option("geomesa.feature", feature)

        .load()

    dataFrame.createOrReplaceTempView(feature)


    Python:

     

    ### This must come first

    import geomesa_pyspark

    conf = geomesa_pyspark.configure(

        jars=['/usr/lib/spark/jars/geomesa-hbase-spark-runtime_2.11-2.3.0.jar'],

        packages=['geomesa_pyspark','pytz'],

        spark_home='/usr/lib/spark/').\

        setAppName('MyTestApp')

    ###


    from pyspark.sql import SparkSession

    spark = ( SparkSession

        .builder

        .config(conf=conf)

        .getOrCreate()

    )


    feature = "orbcomm"

    catalog = "ons-historical"

    hbase = "hbase.optix-ons-local:2181"


    params = { "hbase.zookeepers": hbase, "hbase.catalog": catalog }

    df = spark.read.format("geomesa").options(**params).option("geomesa.feature", feature).load()

    df.createOrReplaceTempView(feature)


    To access the different datasets change the feature name:

    ExactEarth: ee

    Orbcomm: orbcomm

    ADSBx: adsbx