Apache Airflow

Instalación y uso de la librería

Vamos a instalar nuestro Apache Airflow en una máquina virtual usando multipass. Seguimos los pasos correspondientes sobre nuestro máquina, lanzando primero una nueva instancia:

multipass launch --name airapache --cpus 4 --mem 8G
multipass shell airapache
sudo apt update
sudo apt install python3-pip

# Airflow needs a home. `~/airflow` is the default, but you can put it
# somewhere else if you prefer (optional)
export AIRFLOW_HOME=~/airflow

# Install Airflow using the constraints file
AIRFLOW_VERSION=2.3.2

PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

# For example: 3.7
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.3.2/constraints-3.7.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"


# Reboot the VM
exit
multipass restart airapache
multipass shell airapache

Acceso via SSH

Averiguar la IP de nuestras máquinas con multipass

% multipass list
Name                    State             IPv4             Image
airapache                  Running           192.168.64.19     Ubuntu 20.04 LTS
                                          10.1.219.64

Por ejemplo, nuestra IP será 192.168.64.19.

Dentro de nuestra máquina virtual, realizamos los siguientes pasos: - En la MV de Ubuntu, cambiaremos un “no” por un “yes”: sudo nano /etc/ssh/sshd_config Nota: para guardar el documento ctrl+o, y para salir ctrt+x 3fa96db1b6104ea98f8fb51471676347 - Reiniciamos el servicio (¿Qué es un servicio/daemon?) sudo systemctl restart sshd - Añadimos un password a nuestro usuario ```

sudo passwd ubuntu

airflow standalone ```

Desde nuestra máquina Host ya podemos conectarnos:

ssh -L 8080:192.168.64.20:8080 ubuntu@192.168.64.20

Y en cualquier navegador del HOST podemos acceder a la url pertinente: localhost:8080
Finalmente, activamos el ejemplo: example_bash_operator . ¿Qué información nos ofrecen las diferentes pestañas?

Un primer DAG con Apache Airflow

Vamos a realizar ciertos puntos del tutorial de la propia herramienta:

https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html

[1]:

import random as rnd
import time

def my_task1():
    # Esta función es muy compleja, que obtiene un valor después de muchisímo tiempo de ejecución
    try:
        time.sleep(5)
        value = rnd.random()
        return {"value1":value}
    except:
        none

def my_task2():
    # Esta función es muy compleja; pero no tarda tanto.
    try:
        value = rnd.random()
        time.sleep(8)
        return {"value2":value}
    except:
        none


def my_task_max(value1, value2):
    # Esta función muestra como combinar los resultados de otras tareas predecesoras
    valueMax = max(value1["value1"],value2["value2"])
    with open("/tmp/mydata.csv","a+") as f:
        f.write("%.4f,"%valueMax)

[ ]:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import timedelta, datetime

default_args = {
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),

}

dag = DAG('4miPrimerDAG',
     default_args=default_args,
     start_date = datetime(2019,1,1),#datetime.now()-timedelta(minutes=1),
     schedule_interval = timedelta(minutes=2)
    )

[ ]:

t1 = PythonOperator(dag=dag,
        task_id='my_task1',
        python_callable=my_task1)

t2 = PythonOperator(dag=dag,
        task_id='my_task2',
        python_callable=my_task2)


t3 = PythonOperator(dag=dag,
        task_id='my_MAX',
        op_kwargs={'value1': t1.output, 'value2': t2.output},
        python_callable=my_task_max)

[ ]:

# [t1, t2] >> t3

t1.set_downstream(t3)
t2.set_downstream(t3)

[ ]:

# Todo junto

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import timedelta, datetime

import random as rnd
import time

def my_task1():
    # Esta función es muy compleja, que obtiene un valor después de muchisímo tiempo de ejecución
    try:
        time.sleep(5)
        value = rnd.random()
        return {"value1":value}
    except:
        none

def my_task2():
    # Esta función es muy compleja; pero no tarda tanto.
    try:
        value = rnd.random()
        time.sleep(8)
        return {"value2":value}
    except:
        none


def my_task_max(value1, value2):
    # Esta función muestra como combinar los resultados de otras tareas predecesoras
    valueMax = max(value1["value1"],value2["value2"])
    with open("/tmp/mydata.csv","a+") as f:
        f.write("%.4f,"%valueMax)

default_args = {
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),

}

dag = DAG('4miPrimerDAG',
     default_args=default_args,
     start_date = datetime(2019,1,1),#datetime.now()-timedelta(minutes=1),
     schedule_interval = timedelta(minutes=2)
    )

t1 = PythonOperator(dag=dag,
        task_id='my_task1',
        python_callable=my_task1)

t2 = PythonOperator(dag=dag,
        task_id='my_task2',
        python_callable=my_task2)


t3 = PythonOperator(dag=dag,
        task_id='my_MAX',
        op_kwargs={'value1': t1.output, 'value2': t2.output},
        python_callable=my_task_max)


# [t1, t2] >> t3

t1.set_downstream(t3)
t2.set_downstream(t3)

# Recordatorio de algunos comandos
# python3 mydag.py
# airflow db init

# airflow tasks list miPrimerDAG
# airflow tasks test miPrimerDAG my_task1
# airflow dags trigger miPrimerDAG

Actividad del módulo

Analiza diaramente las noticias de ciencia que se publican con el API de Inshorts (ver ejemplo en celdas posteriores). Guarda un registro estadístico de: - Número de noticias - Número de palabras de cada noticia en Content y en title - Registra la hora y fecha de la publicación.

Nota del API: - https://github.com/cyberboysumanjay/Inshorts-News-API - un recopilatorio de APIs: https://github.com/public-apis/public-apis

[3]:

import requests
req = requests.get("https://inshorts.deta.dev/news?category=science")

[10]:

print(req.status_code)

data = dict(req.json())
print(data)

200
{'category': 'science', 'data': [{'author': 'Ankush Verma', 'content': 'NASA has said that the first image from its James Webb Space Telescope is the deepest and sharpest infrared image of the distant universe to date. "Known as Webb\'s First Deep Field, this image of galaxy cluster SMACS 0723 is overflowing with detail. Thousands of galaxies have appeared in Webb\'s view for the first time," NASA added.', 'date': '12 Jul 2022,Tuesday', 'id': '463a2523b39e442187d634a64dbfc9eb', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/12_tue/img_1657593610944_453.jpg?', 'readMoreUrl': 'https://twitter.com/NASAWebb/status/1546621080298835970?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '08:32 am', 'title': 'Deepest image of the early universe ever taken released by NASA', 'url': 'https://www.inshorts.com/en/news/deepest-image-of-the-early-universe-ever-taken-released-by-nasa-1657594971552'}, {'author': 'Apaar Sharma', 'content': 'A Chinese spacecraft has acquired imagery data covering all of Mars, including visuals of its south pole, after circling the planet more than 1,300 times since early last year, state media reported on Wednesday. Other images include photographs of the 4,000-kilometre (2,485-mile) long canyon Valles Marineris, and impact craters of highlands in the north of Mars known as Arabia Terra.', 'date': '29 Jun 2022,Wednesday', 'id': '80a88ec241534f7cb7b690de82a26c59', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/29_wed/img_1656498527637_233.jpg?', 'readMoreUrl': 'https://gallery-repo.inshorts.com/gallery/view/bf27d51d-49a5-40eb-874c-f2713e83368c?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '04:19 pm', 'title': 'In pics: New images of entire Mars captured by Chinese spacecraft', 'url': 'https://www.inshorts.com/en/news/in-pics-new-images-of-entire-mars-captured-by-chinese-spacecraft-1656499797962'}, {'author': 'Apaar Sharma', 'content': "There has been an emergence of a potential Omicron sub-variant referred to as BA.2.75, first reported from India and then from 10 other countries, said the WHO. This sub-variant seems to have a few mutations on the receptor-binding domain of the spike protein, the WHO added. It's too early to know if this sub-variant is clinically more severe, it added.", 'date': '06 Jul 2022,Wednesday', 'id': '2e277fe1c1cd404ab492c227ba584d88', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/6_wed/img_1657076855039_614.jpg?', 'readMoreUrl': 'https://twitter.com/WHO/status/1544413181027778561?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '08:54 am', 'title': 'Potential Omicron sub-variant of COVID-19 reported from India & 10 countries: WHO', 'url': 'https://www.inshorts.com/en/news/potential-omicron-subvariant-of-covid19-reported-from-india-10-countries-who-1657077883080'}, {'author': 'Apaar Sharma', 'content': "Residents of Telangana's Jagtial town witnessed a rare weather phenomenon as fish rained from the sky. 'Rain of animals' is a phenomenon that occurs when small water animals like frogs, crabs, and fish are swept up in waterspouts or drafts that occur on the surface of the earth. They are then rained down at the same time as the rain.", 'date': '10 Jul 2022,Sunday', 'id': '3446933dff8c44f78aa2ec9b6011fe36', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/10_sun/img_1657446750264_30.jpg?', 'readMoreUrl': 'https://news.abplive.com/telangana/telangana-jagtial-witness-rain-of-animals-phenomenon-amid-heavy-rainfall-watch-1541760/amp?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '03:54 pm', 'title': "Rare 'rain of animals' witnessed in Telangana's Jagtial; video surfaces", 'url': 'https://www.inshorts.com/en/news/rare-rain-of-animals-witnessed-in-telanganas-jagtial-video-surfaces-1657448647361'}, {'author': 'Sakshita Khosla', 'content': "The International Space Station (ISS) sustainably released around 172 pounds (78 kg) of waste using a first-of-its-kind specialised trash bag on July 2. The specialised bag has been developed by NASA's Johnson Space Center and Nanoracks and contains filthy garments, foam, packing materials, used office supplies, and hygiene items. The bag will burn up while re-entering Earth's atmosphere.", 'date': '09 Jul 2022,Saturday', 'id': '4ddc13b196eb493ca160e7ae618e440f', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/9_sat/img_1657366542243_790.jpg?', 'readMoreUrl': 'https://tech.hindustantimes.com/amp/tech/news/garbage-on-international-space-station-here-is-how-nasa-throws-it-out-in-space-71657264224108.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '05:20 pm', 'title': '78 kg garbage dropped from the space station for the first time, video released', 'url': 'https://www.inshorts.com/en/news/78-kg-garbage-dropped-from-the-space-station-for-the-first-time-video-released-1657367442930'}, {'author': 'Anmol Sharma', 'content': 'NASA has shared a picture captured by the Fine Guidance Sensor (FGS) of the world\'s biggest telescope, James Webb Space Telescope. "Talk about an overachiever! Gaze at this test image...an unexpected & deep view of the universe," NASA wrote alongside the picture. NASA is expected to release the first full-colour images from the telescope next week. ', 'date': '08 Jul 2022,Friday', 'id': '212c7490880542c98ed70f6df1331c2a', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/8_fri/img_1657272790823_899.jpg?', 'readMoreUrl': 'https://blogs.nasa.gov/webb/2022/07/06/webbs-fine-guidance-sensor-provides-a-preview/?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '03:51 pm', 'title': "NASA shares pic from world's biggest telescope, says 'Unexpected view of the universe'", 'url': 'https://www.inshorts.com/en/news/nasa-shares-pic-from-worlds-biggest-telescope-says-unexpected-view-of-the-universe-1657275670114'}, {'author': 'Anmol Sharma', 'content': "A plant species, Brachystelma attenuatum, which scientists had presumed extinct has been rediscovered in Himachal Pradesh's Hamirpur and Mandi districts after 188 years. It was identified by researchers from Botanical Survey of India, Dehradun and Himachal Pradesh University, Shimla. Its last record dates back to 1835 when British botanists John Royle and Robert Wight found the plant in Himachal.", 'date': '04 Jul 2022,Monday', 'id': 'bcc2e7b1e7eb4b63957b1a905d08f063', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/4_mon/img_1656928554462_431.jpg?', 'readMoreUrl': 'https://www.hindustantimes.com/cities/chandigarh-news/plant-presumed-extinct-rediscovered-after-188-years-in-himachal-101656882836396-amp.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '03:40 pm', 'title': 'Plant presumed extinct rediscovered after 188 years in Himachal Pradesh', 'url': 'https://www.inshorts.com/en/news/plant-presumed-extinct-rediscovered-after-188-years-in-himachal-pradesh-1656929427684'}, {'author': 'Anmol Sharma', 'content': 'New Zealanders across the North Island reported a mysterious fireball and a huge flash of light streaking across the sky on Thursday afternoon. They also reported rumbling and crackling sounds. A plumber named Curtis Powell captured the phenomenon on his dashcam while driving. He called it a "once-in-a-lifetime" spectacle.', 'date': '08 Jul 2022,Friday', 'id': '96cc5e49e1f3445aab3624a5b609bc0e', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/8_fri/img_1657249554385_971.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/news/world/australasia/wellington-rare-meteor-sighting-b2117929.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '09:00 am', 'title': 'Mysterious fireball spotted over New Zealand, explosions heard; pic surfaces', 'url': 'https://www.inshorts.com/en/news/mysterious-fireball-spotted-over-new-zealand-explosions-heard-pic-surfaces-1657251039949'}, {'author': 'Ankush Verma', 'content': 'NASA has asked for public\'s help in spotting clouds on Mars. The space agency has organised a project, called "Cloudspotting on Mars", that uses its citizen science platform Zooniverse. The information may help researchers figure out why Mars\' atmosphere is just 1% as dense as Earth\'s even though ample evidence suggests the planet used to have a much thicker atmosphere.', 'date': '30 Jun 2022,Thursday', 'id': '49bfdef9b3a04d54bee4795bbd523231', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/30_thu/img_1656579786383_268.jpg?', 'readMoreUrl': 'https://twitter.com/NASA/status/1541914753785905157?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '02:47 pm', 'title': "NASA asks for public's help in spotting clouds on Mars, shares link", 'url': 'https://www.inshorts.com/en/news/nasa-asks-for-publics-help-in-spotting-clouds-on-mars-shares-link-1656580630826'}, {'author': 'Sakshita Khosla', 'content': 'The fossilised brain of a creature with three eyes, that lived 500 million years ago is being studied by scientists. Pictures of the remains of the brain and nerves of the marine predator called Stanleycaris have been released. "The details are so clear it\'s as if we\'re looking at an animal that died yesterday," said a researcher on the fossil.', 'date': '11 Jul 2022,Monday', 'id': '112e15b04da04f05a0f8be73d65c03d9', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/11_mon/img_1657533473276_807.jpg?', 'readMoreUrl': 'https://twitter.com/ROMtoronto/status/1545422918259204097?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '03:44 pm', 'title': 'Pic of 500-million-year-old fossilised brain of 3-eyed predator released', 'url': 'https://www.inshorts.com/en/news/pic-of-500millionyearold-fossilised-brain-of-3eyed-predator-released-1657534441719'}, {'author': 'Arshiya Chopra', 'content': 'The Chandrayaan-2 mission of the Indian Space Research Organisation (ISRO) has captured the site of impact on the moon after a used rocket hit the lunar surface. It shared before and after images of the lunar surface to show the difference after the impact. "The impact created a double crater which is 28 metre wide," ISRO said. ', 'date': '12 Jul 2022,Tuesday', 'id': 'c926cdae17df445aa0936b34ff5f2cbe', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/12_tue/img_1657602788502_623.jpg?', 'readMoreUrl': 'https://www.news18.com/amp/news/buzz/isro-photos-show-double-craters-created-by-rocket-hitting-moon-surface-5533975.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '11:01 am', 'title': 'ISRO shares photos of double craters created by rocket hitting the moon', 'url': 'https://www.inshorts.com/en/news/isro-shares-photos-of-double-craters-created-by-rocket-hitting-the-moon-1657603882463'}, {'author': 'Ankush Verma', 'content': 'Researchers have discovered "never-seen-before" carbon crystals in dust from a space rock that had exploded in Russia nearly a decade ago. The exploding space rock, known as a superbolide, fell on 15 February 2013, and is the biggest meteoroid to have exploded on Earth in the 21st century to date. Researchers published the study in EPJ Plus journal.', 'date': '05 Jul 2022,Tuesday', 'id': 'ae030c0b886247158d866902311fcc09', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/5_tue/img_1657012062025_994.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/space/exotic-crystals-biggest-asteroid-century-b2115801.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '02:52 pm', 'title': "'Never-seen-before' crystals found in 21st century's biggest meteoroid to fall on Earth", 'url': 'https://www.inshorts.com/en/news/neverseenbefore-crystals-found-in-21st-centurys-biggest-meteoroid-to-fall-on-earth-1657012969415'}, {'author': 'Anmol Sharma', 'content': 'Deep-sea scientists have discovered a rare transparent fish during a routine survey in Alaska. Sharing a picture, Sarah Friedman, a fish biologist at National Oceanic and Atmospheric Administration (NOAA), tweeted, "Been hoping to see one of these in person for a long time! Blotched snailfish (Crystallichthys cyclospilus)." The blotched snailfish camouflage themselves to be invisible to predators.', 'date': '03 Jul 2022,Sunday', 'id': '67601a12048b4d5fba1034fa6da1c482', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/3_sun/img_1656824795215_327.jpg?', 'readMoreUrl': 'https://twitter.com/sarahtfried/status/1538596561738903552?s=20&t=dAsc6Vvq4aZA1sGp47JzIQ&utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '10:53 am', 'title': 'Rare transparent fish found in Alaska, pic surfaces', 'url': 'https://www.inshorts.com/en/news/rare-transparent-fish-found-in-alaska-pic-surfaces-1656825796621'}, {'author': 'Sakshita Khosla', 'content': 'NASA has said that the impact from a "mystery rocket body" on the Moon has created two new unusual craters on Earth\'s natural satellite. It released pictures of the impact site, which has two craters superimposed on each other. The unknown rocket body was spotted heading towards a lunar collision late last year and the impact occurred on March 4.', 'date': '28 Jun 2022,Tuesday', 'id': '78abb890d98f49d98e250d94482bdd7d', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/28_tue/img_1656420031839_384.jpg?', 'readMoreUrl': None, 'time': '06:43 pm', 'title': "'Mystery' rocket crashes on the Moon, creating 2 new craters; pic released", 'url': 'https://www.inshorts.com/en/news/mystery-rocket-crashes-on-the-moon-creating-2-new-craters-pic-released-1656422032121'}, {'author': 'Daisy Mowke', 'content': "A new Moon occurs when all of Sun's light is reflected away from Earth, and the side of Moon facing Earth is barely visible. Sometimes the dark face of Moon catches Earth's reflected glow and returns that light. This phenomenon is called earthshine. The earthshine Moon will be visible for next few days but will be brightest over the weekend.", 'date': '01 Jul 2022,Friday', 'id': '8895709b9c0a4efe8244aa87ef52e0d2', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/1_fri/img_1656684700712_923.jpg?', 'readMoreUrl': 'https://tech.hindustantimes.com/amp/tech/news/earthshine-moon-2022-know-when-and-how-to-watch-this-spectacular-event-71656659906847.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '08:03 pm', 'title': 'What is earthshine, the phenomenon that will make the Moon glow this weekend?', 'url': 'https://www.inshorts.com/en/news/what-is-earthshine-the-phenomenon-that-will-make-the-moon-glow-this-weekend-1656686014285'}, {'author': 'Anmol Sharma', 'content': "A meteor was recently spotted causing huge flashlight through the night sky over Chile's Santiago. Academics at the Concepción University said that it was a small rock body that burned up upon entering the Earth’s atmosphere. According to local media, the bright flashes of light were accompanied with loud booming sounds.", 'date': '11 Jul 2022,Monday', 'id': '50d3b64b1582477c887455ec1ad179b8', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/11_mon/img_1657539276107_931.jpg?', 'readMoreUrl': 'https://www.news18.com/amp/news/buzz/caught-on-camera-meteor-lights-up-the-night-sky-over-chiles-santiago-5531773.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '05:41 pm', 'title': "Huge flashlight caused by meteor seen in Chile's night sky, pic goes viral", 'url': 'https://www.inshorts.com/en/news/huge-flashlight-caused-by-meteor-seen-in-chiles-night-sky-pic-goes-viral-1657541464267'}, {'author': 'Swati Dubey', 'content': 'As many as 19 (58%) out of 33 women who underwent a uterus transplant in the US between 2016 and 2021 have delivered a total of 21 babies, researchers said. All the babies were delivered by Cesarean section, 14 months after the transplant on average. More than half of these babies were born after 36 weeks of gestation. \n', 'date': '07 Jul 2022,Thursday', 'id': 'c97bfcadcb5548c49dd88ca0bbb5b20a', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/7_thu/img_1657188105923_861.jpg?', 'readMoreUrl': 'https://www.reuters.com/business/healthcare-pharmaceuticals/uterus-transplants-allow-successful-pregnancies-us-women-study-2022-07-06/?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '04:45 pm', 'title': 'Uterus transplants allow successful pregnancies in US women: Study', 'url': 'https://www.inshorts.com/en/news/uterus-transplants-allow-successful-pregnancies-in-us-women-study-1657192533642'}, {'author': 'Swati Dubey', 'content': "Rainforest chimpanzees of Uganda's Waibira community have started digging small wells after apparently learning the skill from an immigrant female chimpanzee named Onyofi. Researchers said it's the first time that well-digging has been observed in rainforest chimpanzees. However, only female chimps dug the wells and no adult males were seen doing so, although they used the wells for water.\n", 'date': '29 Jun 2022,Wednesday', 'id': '7306e54ce6c2409ab6aec6fa326303be', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/29_wed/img_1656496233157_892.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/climate-change/news/chimpanzees-uganda-behaviour-study-b2111106.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '04:28 pm', 'title': 'Rainforest chimps digging wells after learning from immigrant ape: Research', 'url': 'https://www.inshorts.com/en/news/rainforest-chimps-digging-wells-after-learning-from-immigrant-ape-research-1656500325645'}, {'author': 'Swati Dubey', 'content': "A giant fossil of the ancestor of kangaroos that lived over 42,000 years ago was found in Papua New Guinea. The squat and muscular animal, that lived in diverse montane rainforest, evolved to eat tough leaves from trees and shrubs using thick jaw bone and strong chewing muscles, researchers said. It, however, didn't resemble its present-day Australian descendants, they added.\n", 'date': '30 Jun 2022,Thursday', 'id': 'e85add6385574a6db10dcbd70729352a', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/30_thu/img_1656581809004_916.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/news/science/giant-kangaroo-papua-new-guinea-b2111735.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '04:59 pm', 'title': 'Giant fossil of kangaroo species that lived over 42,000 yrs ago found', 'url': 'https://www.inshorts.com/en/news/giant-fossil-of-kangaroo-species-that-lived-over-42000-yrs-ago-found-1656588547401'}, {'author': 'Swati Dubey', 'content': "NASA's CAPSTONE spacecraft to the moon was launched from New Zealand on Tuesday. The microwave oven-sized spacecraft will study a specific orbit and verify if it's suitable for a space station that NASA aims to launch later this decade. It wants to build a station for astronauts in orbit where they can stop before and after going to the moon.\n", 'date': '29 Jun 2022,Wednesday', 'id': 'f1fb94542d6b4df9a33bc09ab1d7d535', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/29_wed/img_1656489561671_301.jpg?', 'readMoreUrl': None, 'time': '02:36 pm', 'title': 'NASA’s CAPSTONE mission launches to the moon', 'url': 'https://www.inshorts.com/en/news/nasas-capstone-mission-launches-to-the-moon-1656493605945'}, {'author': 'Swati Dubey', 'content': "Scientists have discovered the world's largest waterlily belonging to the genus Victoria at Kew Gardens. Named V boliviana, in honour of Bolivian partners in the research, leaves of the newly discovered waterlily species grow up to three metres. It has flowers which turn from white to pink and bear spiny petioles. It's one of the world's botanical wonders, scientists said.\n", 'date': '05 Jul 2022,Tuesday', 'id': '4c3b14b8c3de4413849ea9f0c6e50aea', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/5_tue/img_1657009575268_429.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/climate-change/news/giant-waterlily-kew-botanical-wonder-b2115052.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '02:55 pm', 'title': "World's largest waterlily species discovered in Kew Gardens", 'url': 'https://www.inshorts.com/en/news/worlds-largest-waterlily-species-discovered-in-kew-gardens-1657013154254'}, {'author': 'Swati Dubey', 'content': 'Scientists have discovered nearly 1,000 new species of microbes in the frozen Tibetan Plateau. Almost 98% of these 968 bacteria species are yet unknown to science, some being over 15,000 years old. The study stated that when the glaciers melt, these microorganisms, possibly carrying novel virulence factors, could lead to local epidemics or pandemics in India and China.\n', 'date': '05 Jul 2022,Tuesday', 'id': '8fb12cf336794a4980b94652f3245b27', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/5_tue/img_1657007540996_148.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/climate-change/news/tibet-glaciers-new-bacteria-melting-b2115035.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '02:33 pm', 'title': 'Nearly 1,000 unknown microbes found trapped in Tibetan glaciers', 'url': 'https://www.inshorts.com/en/news/nearly-1000-unknown-microbes-found-trapped-in-tibetan-glaciers-1657011793011'}, {'author': 'Swati Dubey', 'content': 'Scientists have developed an artificial photosynthesis method to grow food without sunlight. They used a 2-step electrocatalytic process to convert CO2, electricity and water into acetate. This acetate was consumed by food-producing organisms to grow in dark. When combined with solar panels, the hybrid organic-inorganic system could increase conversion efficiency of sunlight into food by 18 times.', 'date': '30 Jun 2022,Thursday', 'id': 'e99561b47f7f4230b8a9825f59c416ee', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/06_jun/30_thu/img_1656577668636_884.jpg?', 'readMoreUrl': None, 'time': '04:02 pm', 'title': 'Scientists use artificial photosynthesis to grow food in the dark', 'url': 'https://www.inshorts.com/en/news/scientists-use-artificial-photosynthesis-to-grow-food-in-the-dark-1656585173008'}, {'author': 'Swati Dubey', 'content': "HIV has an early and substantial impact on the process of ageing, a study has found. Highlighting the need for early diagnosis of the infection, the study stated that HIV can start impacting ageing within just two to three years of contracting the virus. Moreover, HIV may rapidly cut about five years off an individual's life span, it stated.", 'date': '02 Jul 2022,Saturday', 'id': '9ec26a1054d54624b1e0da169751b6d4', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/2_sat/img_1656756679148_467.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/news/science/hiv-test-infection-speed-up-ageing-b2113503.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '04:44 pm', 'title': 'HIV speeds up body’s ageing process soon after infection: Study', 'url': 'https://www.inshorts.com/en/news/hiv-speeds-up-bodys-ageing-process-soon-after-infection-study-1656760494379'}, {'author': 'Swati Dubey', 'content': "Mosquitoes choose their target on basis of people's body temperature, odour and CO2 from their breath. Some viruses alter body odour in a way that makes people more attractive to mosquitoes, a study stated. They can also transfer the virus from an infected person to another person through bites. Researchers also found that mosquitoes are more attracted to acetophenone's odour.", 'date': '02 Jul 2022,Saturday', 'id': '5d5b5f8a70fe40609044395321a91a59', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/2_sat/img_1656753507312_907.jpg?', 'readMoreUrl': 'https://www.independent.co.uk/news/science/viruses-mosquitoes-study-bacteria-smell-b2113333.html?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '03:56 pm', 'title': 'Viruses make people more attractive to mosquitoes: Study', 'url': 'https://www.inshorts.com/en/news/viruses-make-people-more-attractive-to-mosquitoes-study-1656757596386'}], 'success': True}

[12]:

news = data["data"]
print(len(news))
print(news[0])

25
{'author': 'Ankush Verma', 'content': 'NASA has said that the first image from its James Webb Space Telescope is the deepest and sharpest infrared image of the distant universe to date. "Known as Webb\'s First Deep Field, this image of galaxy cluster SMACS 0723 is overflowing with detail. Thousands of galaxies have appeared in Webb\'s view for the first time," NASA added.', 'date': '12 Jul 2022,Tuesday', 'id': '463a2523b39e442187d634a64dbfc9eb', 'imageUrl': 'https://static.inshorts.com/inshorts/images/v1/variants/jpg/m/2022/07_jul/12_tue/img_1657593610944_453.jpg?', 'readMoreUrl': 'https://twitter.com/NASAWebb/status/1546621080298835970?utm_campaign=fullarticle&utm_medium=referral&utm_source=inshorts ', 'time': '08:32 am', 'title': 'Deepest image of the early universe ever taken released by NASA', 'url': 'https://www.inshorts.com/en/news/deepest-image-of-the-early-universe-ever-taken-released-by-nasa-1657594971552'}

Referencias

Data Pipelines with Apache Airflow. Manning Publications Bas P. Harenslak, Julian Rutger de Ruiter. 2021
https://betterdatascience.com/apache-airflow-run-tasks-in-parallel/