Click > to expand page tree items
Link to the PETs track main competition:
About the PETs track
The Privacy Enhancing Technologies Track of the UN Datathon is fully hosted on the Antigranular platform powered by Oblivious. It empowers participants to champion responsible data science to drive positive social change using Privacy-Enhancing Technologies inside Private Python.
PETs represent a range of methods, systems, and tools geared towards the secure handling of sensitive data. They maintain individual privacy while facilitating critical data processing tasks. PETs also ensure adherence to the most stringent data protection policies and regulations.
The Private Dataset source
Participants will analyse the data that delves into the impact of shocks on agricultural livelihoods using the Data in Emergencies (DIEM) Information System by the Food and Agriculture Organization of the United Nations (FAO).
DIEM amasses continuously refreshed data on agricultural impacts from crises across 30+ countries, reaching 150,000 households annually through 37 surveys. This data includes information across five categories—income, crops, livestock, food security, and aid needs.
The Objective of the PETs Track
The objective of DIEM is to understand the impact of shocks in food crisis contexts and inform decision-making in support of agricultural livelihoods. The data fuels real-time, stratified analyses vital for bolstering agricultural resilience.
The data can be utilised to monitor food security trends, analyse the impact of shocks, take stock of food security and livelihoods, develop programs, mobilise support partners, see how different situations change over time and across countries to influence policy processes at the national and regional level, and more.
The objective of this Datathon is for participants to draw insights from the sensitive data in a privacy-preserving manner to develop innovative solutions and fuel informed decisions that would benefit and make a real difference in the lives of the impacted communities.
How to Get Started?
To make sure that you have a fantastic experience with the UN Datathon, we have the following recommendations:
- Test that your login to the Antigranular platform is working and you are assigned to the right team.
- Once logged in, you can get ready for the PETs Track of the UN Datathon by exploring the PETs track competition and the UN Datathon Wiki.
- Start to explore the PETs track dataset and see the FAO Data details for the PETs track
- Follow this tutorial on how to interact with the PETs track secure enclave
- Continue learning about Private Python and Responsible Data Science by watching the recordings of the Oblivious Office Hours
- Check out the example notebooks to deep dive into Differentially private machine learning
- To learn more about PETs and how Antigranular works, read our documentation
How to Handle Private Data through Antigranular?
The data can only be accessed using both input and output PETs that guarantee their privacy. We implement secure enclaves that keep the data in a Trusted Execution Environment as well as apply differential privacy to the outputs to warrant original inputs cannot be reverse-engineered. This means:
- The data remains opaque at runtime and allows extremely limited input and output
- A cryptographic handshake creates a secure, undisclosed channel of communication from a computer to the enclave
- We ensure end-to-end security, where the data stays invisible to anyone outside of the enclave
Analysing the Data
- Sign Up and Log In with your email credentials.
- Navigate to the UN Datathon competition in Antigranular.
- Open a new Jupyter Notebook using any available option, including Google Colab or your local system.
- Install the Antigranular package by typing in a "pip install antigranular" command.
- Connect to the secure enclave by copying the code block at the top right corner of the competition page which includes your custom credentials. Paste them into your Jupyter Notebook to connect to the secure enclave.
- Upon successful login, you'll see the session ID and the %%ag cell magic registered to your system.
- The datasets need to be loaded using the load_dataset method. The dataset has been sub-divided into 19 subsets and you can import each of these separately into your coding environment. Google Sheet containing details of each data subset, data export instructions, column names and descriptions.
- You are now ready to analyse the data, make predictions, and flex your skills using %%ag magic remote execution. Check out the sample notebooks and details of supported packages for more examples.
- Antigranular is not just about accuracy but also about using the least amount of privacy budget. Navigate the trade-off like a boss to come out on top. Head to the Antigranular docs for details of the scoring system and best epsilon practice.
Making a Submission
Please include your python code, or notebook in your submission.
Please make sure that you are sharing the repository link showing your explorations.
For complete information on how to make a submission, visit the UN Datathon Wiki page.
Why Responsible Data Science?
Data scientists rarely have access to the most sensitive data sources and when they do, they often get access far below the level of granularity required to do their job.
Implementing PETs enables data scientists to safely utilise sensitive data that would otherwise remain inaccessible. It empowers you to unlock the full potential of the world’s most impactful data without compromising on privacy or security of individuals.
A wealth of academic research is dedicated to privacy-enhancing technologies, yet their adoption in practical applications lags behind. We want to change this by integrating PETs seamlessly into how every data scientist and developer already works with Antigranular. By providing a platform for the application of PETs, we strive to stimulate their widespread use.
About Antigranular’s Private Python Package
Antigranular is a community-driven, open-source platform developed by Oblivious that merges confidential computing and differential privacy. This creates a secure environment for handling unseen confidential data.
Antigranular enables data scientists to work with the latest PETs using our private python and toggle between confidential compute and regular code blocks within their Jupyter Notebook.
Antigranular’s private python is a specialised version of the Python programming language that offers a user-friendly and accessible coding environment. It is designed specifically for working with differentially private data, ensuring privacy protections are embedded at every step.
Start the hackathon:
Access the data: https://www.antigranular.com/datasets
Access the competition: https://www.antigranular.com/competitions
Logging into the secure environment: https://youtu.be/noWf1UaHn1w?si=L1dksWelLfTIN6L1
Read the docs:
Antigranular docs for private python: https://docs.antigranular.com/
Ask for help:
UN Datathon discord: https://discord.gg/2RX48akAaB
Oblivious discord: https://discord.gg/8vVhY3ENMW
- Introduction to the Antigranular platform
- Oblivious Office Hours
- Using private python with Antigranular
In recognition of the need to better understand the impact of shocks on agricultural livelihoods in food crisis contexts, the Food and Agriculture Organization of the United Nations (FAO) established the Data in Emergencies Information System. Driven by regularly collected primary data in food crisis countries, its objective is to inform decision-making in support of agricultural livelihoods in fragile and shock-prone environments. Since the launch of the DIEM Hub in June 2020, DIEM surveys have been completed in over 30 countries reaching approximately 150 000 households per year. At the center of the DIEM Information System is the DIEM-Monitoring System which performs regular, standardized and frequent household surveys.
FAO Data in Emergencies Hub page on the Datathon Wiki
- No labels