Censys Universal Internet Dataset

The Censys Universal Internet Dataset is the most comprehensive Internet-wide scan data in the industry. This dataset supersedes the legacy IPv4 and IPv4 Banners datasets.

Censys global scanning engine continuously walks the entire IPv4 space, detecting 107 protocols on over 3500 ports to produce visibility in a high-resolution map of the public Internet. This provides a significant tool for threat hunters, attack surface managers, and other security professionals.

Every day, a 'snapshot' of every service on the Internet is taken and added as a partition to a dataset available to Censys Enterprise customers, who can query this data using Google BigQuery.

Scan data in any snapshot is never more than 10 days old, and the data for a service is either structured protocol-specific key-value pairs or, if the service is using an unknown protocol, banner data and/or TLS information.

Usage

BigQuery Dataset Names

• The entire history of daily Internet snapshots:
  censys-io.universal_internet_dataset_v2
• Today’s snapshot of the live Internet:
  censys-io.universal_internet_dataset_v2.current
• A snapshot from a specific day:
  censys-io.universal_internet_dataset_v2.{yyyyMMdd}

Data model

Read this introduction to the host data model to learn how Censys represents Internet hosts.

Schema

Explore the Censys Universal Internet Dataset schema for fields you can query.

Query Syntax

Use Standard SQL to query the Censys datasets, which is now the default for Google BigQuery.

Examples

Want to know the port distribution of SMTP services using the compromised TLSv1_0 encryption method on Jun. 9, 2021? Try this query:

SELECT
  tls.version_selected,
  service_name,
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.20210609`,
  UNNEST(services)
WHERE
  tls.version_selected = 'TLSv1_0'
  AND service_name = 'SMTP'
GROUP BY
  1,
  2,
  3
ORDER BY
  4 DESC

Want to see the software (in CPE format) reported by Telnet services across all ports today? Try running this query:

SELECT
  software.vendor,
  software.product,
  software.version,
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.current`,
  UNNEST(services),
  UNNEST(software) AS software
WHERE
  service_name = 'TELNET'
GROUP BY
  1,2,3,4
ORDER BY
  5 DESC

Or see a count of all the Kubernetes services across the Internet on May 25, 2021, grouped by the port they’re running on:

SELECT
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.20210525`,
  UNNEST(services)
WHERE
  service_name = 'KUBERNETES'
GROUP BY
  1
ORDER BY
  2 DESC