Where portfolio data comes from, and what you may do with it

The question nobody asks until it is a problem

You need data for a project. The instinct is to take some. Three questions decide whether that is fine, awkward, or career-ending: where did it come from, does it describe identifiable people, and what does its licence permit.

Where good data actually lives

• National statistics and open-government portals. India’s data.gov.in, the UK’s data.gov.uk, the EU portal, the US data.gov, Nigeria’s and Kenya’s open data initiatives, Brazil’s dados.gov.br. Census, health, transport, agriculture, budgets, procurement. Usually the most interesting data available anywhere, and almost nobody uses it, which is exactly why a project built on it stands out.
• International bodies. World Bank, WHO, UN, FAO, OECD. Cleanly licensed and well documented.
• Research repositories. UCI, Zenodo, OpenML, Hugging Face Datasets, Papers with Code.
• Kaggle. Enormous and convenient, with a caveat below.
• Your own. Data you generate, log or collect with consent. The strongest option for a portfolio, because nobody else has it.

Licences, in five minutes

The licence is a permission, not a formality:

• CC0 / public domain: do anything.
• CC-BY: do anything, credit the source.
• CC-BY-SA / ODbL: as above, but derived datasets must carry the same licence. OpenStreetMap is ODbL, and this catches people.
• Non-commercial (CC-BY-NC): fine for a portfolio, not fine inside a product.
• ”Research use only”: common on academic sets, and it means what it says.
• No licence at all: the default is that you have no permission. Absence of a licence is not permission.

The Kaggle caveat: many datasets there were uploaded by somebody who was not the rights holder, and the licence field was chosen by that uploader. A dataset marked CC0 that is plainly a scrape of a commercial site is not CC0. Trace it to the original source when the provenance looks thin.

Scraping: the honest state of it

Technically easy. Legally unsettled, and unsettled differently in different countries. The shape of the disagreement, which is the most anybody can honestly give you:

• Access to publicly available pages has been treated relatively permissively in some United States computer-misuse rulings, while a site’s terms of service are a separate contractual question that those rulings did not resolve.
• The European Union has a database right that has no American equivalent, and separate text-and-data-mining provisions with an opt-out for commercial use. Japan and Singapore have broader mining exceptions.
• Copyright in AI training data is being litigated in several countries right now, with decisions pointing in different directions. Anybody who tells you this is settled is describing their preference.

None of this is legal advice, and this course cannot give any. What is defensible practice regardless of jurisdiction: respect robots.txt, rate-limit yourself, prefer a documented API where one exists, never take personal data, never redistribute the raw scrape, and be able to explain in one paragraph why you believed you were entitled to the data. If a project cannot survive that paragraph, choose different data; there is plenty.

Personal data is regulated even when it is public

If it identifies a person, a data protection law probably applies: the GDPR in Europe, India’s Digital Personal Data Protection Act, Nigeria’s Data Protection Act, Brazil’s LGPD, and others. “It was on a public profile” is not an exemption in any of them.

For a portfolio there is no upside. Aggregate, anonymise properly, or pick a different subject.

Never your employer’s data. Not once.

This ends careers, and the mechanism is worth stating plainly. The data is not yours; it is covered by your employment contract, usually by a customer contract underneath that, and frequently by regulation on top. Publishing a chart built from it can be a breach of all three at once, and a reviewer who spots it will assume you would do the same to them.

What to do instead: describe the problem and the method, and reproduce the shape of the data synthetically. Faker, SDV and twenty lines of NumPy generate a table with the same columns, cardinalities and rough distributions. Synthetic data is excellent for demonstrating a pipeline, a model or an interface. It is worthless for demonstrating a finding, and you should say which of the two you are showing.

The habit that costs one minute

Every project gets a DATA.md with the source URL, the licence, the download date and one line on any preprocessing. It takes a minute, it answers the reviewer’s question before they ask it, and in a field where provenance is the live argument of the decade, it marks you as somebody who thinks about it.