Evaluating the Effectiveness ofAutomatic Image Captioning for Web Accessibility
Maurizio Leotta, Fabrizio Mori, Marina Ribaudo
Abstract
Nowadays, the web is a fundamental tool for carrying out many activities spanning from education to work and private life. For this reason, it must be accessible to every user regardless of any form of impairment or disability. Images on the web are a primary means for communicating information and specific HTML constructs have been defined to enrich images with textual descriptions that screen readers can read aloud or can be rendered as Braille.
A relevant problem is that adding a text to describe the content of each image published on a website is a demanding task that requires a non-negligible effort for web developers.
Recently several tools emerged based on machine learning, which can automatically return descriptions for the images provided as input.
In this work, we evaluate the correctness of these generated descriptions by comparing them with a human-defined reference. More in detail, we selected 60 images from Wikipedia and their corresponding descriptions as defined by Wikipedia contributors. These images represent different subjects: human beings, landmarks, and everyday life. We generated the corresponding descriptions employing four state of the practice tools, namely Azure Computer Vision Engine, Amazon Rekognition,Cloudsight, and Auto Alt-Text for Google Chrome. Then, 76 computer science students blindly evaluated the perceived correctness of the descriptions without being aware of their source.
The results show that the descriptions available in Wikipedia are still perceived as the best ones. However, some tools can generate excellent results, and they can be seen as good candidates for the automated and massive addition of image descriptions to websites. This would help to increase the accessibility level of the web drastically.
Dataset of 60 images and corresponding descriptions:
| Category | Image (from Wikipedia) | Wikipedia | Azure Computer Vision Engine | Amazon Rekognition | Cloudsight | Alt-Text for Google Chrome |
| Human - Famous Painting |  | Shakespeare | a man looking at the camera | painting, art, person | man in black and white collared shirt | a man in a suit and tie is smiling |
| Human - Famous Painting |  | portrait of Giulio Cesare | a man wearing a suit and tie | head, sculpture, art, statue, person | mans face concrete statue | a man in a suit and tie is smiling |
| Human - Famous Painting |  | Francesco Melzi - Portrait of Leonardo | a close up of a mans face | skin, wood, art | womans face with blue hair | a close up of a stuffed animal on a bed |
| Human - Famous Painting |  | head and chest side portrait of Dante in red and white coat and cowl | a person wearing a costume | art | woman in red dress painting | a woman wearing a red hat |
| Human - Famous Painting |  | Depiction of Virgil | a drawing of a person | art, drawing, person, sketch | grayscale photo of woman in white shirt | a black and white photo of a man |
| Human - Famous 1900 |  | John Lennon 1969 | John Lennon in glasses looking at the camera | face, person, accessories, glasses, beard | man in black framed eyeglasses | a man with a tie |
| Human - Famous 1900 |  | Walt Disney 1946 | Walt Disney looking at the camera | face, person, head, Walt Disney | man in black and white checkered button up shirt | a man in a suit and tie is smiling |
| Human - Famous 1900 |  | Thomas Edison | Thomas Edison wearing a suit and tie | clothing, chair, furniture, coat, overcoat, suit, person, couch | man in black suit jacket | a man in a suit and tie sitting in a chair |
| Human - Famous 1900 |  | Alan Turing Aged 16 | Alan Turing wearing a suit and tie | tie, accessories, overcoat, coat, suit, clothing, person, Alan Turing | man in black suit jacket | a black and white photo of a man in a suit and tie |
| Human - Famous 1900 |  | Frida Kahlo, by Guillermo Kahlo | Frida Kahlo posing for the camera | face, person, female, Frida Kahlo | grayscale photo of woman in black and white checkered scarf | a woman in a dress shirt and tie |
| Human - Famous 2000 |  | Kylie Jenner March 2020 | Kylie Jenner posing for the camera | face, person, Kylie Jenner | woman with blonde hair and pink lipstick | a woman wearing a blue shirt and a red tie |
| Human - Famous 2000 |  | Kanye West at the 2009 Tribeca Film Festival | Kanye West wearing a suit and tie smiling at the camera | person, face, Kanye West | man in black suit jacket | a man in a suit and tie is smiling |
| Human - Famous 2000 |  | Cristiano Ronaldo in 2018 | a close up of Cristiano Ronaldo wearing a blue shirt | person, face, Cristiano Ronaldo | man in red v neck shirt | a man in a baseball uniform holding a baseball bat |
| Human - Famous 2000 |  | Roger Federer (26 June 2009, Wimbledon) | Roger Federer is swinging a racket at a ball | person, sport, tennis, racket, tennis racket | man in white polo shirt and white pants holding golf club | a man holding a tennis racquet on a tennis court |
| Human - Famous 2000 |  | Lionel Messi in 2018 | a man wearing a blue shirt | face, person, beard | man in white and blue adidas jersey shirt | a man in a suit and tie is smiling |
| Human - Famous 2000 |  | Xi Jinping 2016 | Xi Jinping wearing a suit and tie | tie, accessories, suit, face, person, necktie, Xi Jinping | man in black suit jacket | a man in a suit and tie standing in a room |
| Human - Famous 2000 |  | Vladimir Putin | Vladimir Putin wearing a suit and tie | tie, accessories, person, face, clothing, suit, head, Vladimir Putin | man in black suit jacket | a man in a suit and tie is smiling |
| Human - Famous 2000 |  | Donald Trump official portrait | Donald Trump wearing a suit and tie | accessories, tie, suit, clothing, person, symbol, flag | man in blue suit smiling | a man in a suit and tie is smiling |
| Human - Famous 2000 |  | Angela Merkel - July 2010 | Angela Merkel posing for the camera | person, face, accessories, jewelry, necklace, Angela Merkel | woman in red crew neck shirt | a man wearing a tie and a hat |
| Human - Famous 2000 |  | Jeff Bezos 2016 | Jeff Bezos wearing a suit and tie | accessories, tie, head, clothing, suit, person, face, Jeff Bezos | man in blue suit jacket | a man in a suit and tie is smiling |
| Landmark |  | Eiffel tower and Mars field | a view of a city | outdoors, nature, landscape, city, building, urban, metropolis, scenery, architecture, tower, spire, downtown, high rise | aerial view of eiffel tower during daytime | a large clock tower towering over a city |
| Landmark |  | Beijing CBD Skyline | a river running through a city | city, urban, metropolis, building, high rise, downtown, architecture | city skyline under blue sky during daytime | a large clock tower towering over a city |
| Landmark |  | Towers on the Nile | a large body of water with a city in the background | building, office building, metropolis, urban, city, high rise | high rise buildings near body of water during daytime | a large clock tower towering over a city |
| Landmark |  | Lotus Delhi | a large building with a blue umbrella with Lotus Temple in the background | building, architecture, opera house | white concrete building during night time | a large building with a large clock on it |
| Landmark |  | Istanbul 2010 | a group of people in a large body of water | landscape, outdoors, nature, scenery, aerial view, city, urban, metropolis, building | aerial view of city buildings during daytime | a large group of people on a beach |
| Landmark |  | Jakarta Indonesia Jakarta-History-Museum | a group of people in front of Jakarta History Museum | city, urban, downtown, building, architecture, person | people sitting on bench near brown and white concrete building during daytime | a group of people walking down a street |
| Landmark |  | Mar-elias | a castle like building with a mountain in the background | architecture, housing, monastery, building, landscape, outdoors, nature, scenery, panoramic, urban | brown concrete building on top of hill | a large building with a clock tower on top |
| Landmark |  | 2014 Victoria Island Lagos Nigeria | a tall building in a city | urban, city, building, high rise, metropolis, downtown, architecture | city buildings under white sky during daytime | a city street filled with lots of traffic |
| Landmark |  | London Eye 2009 | a large ship in a body of water with London Eye in the background | amusement park, ferris wheel | ferris wheel near body of water during daytime | a plane flying over a body of water |
| Landmark |  | final circulation of the Kaaba | a view of a rocky mountain with Abraj Al Bait in the background | building, architecture, outdoors, nature, landscape, scenery, tower | aerial view of city buildings during daytime | a large clock tower towering over a city |
| Landmark |  | Red square Moscow cityscape | a group of people standing in front of a building | building, city, urban, downtown, architecture, tower, person, town square, spire | people walking on street near brown concrete building under white clouds and blue sky during daytime | a crowd of people standing in front of a tall building |
| Landmark |  | Mumbai skyline BWSL | a large bridge over a body of water | building, city, urban, metropolis, high rise, bridge | white and blue boat on sea near city buildings during daytime | a large bridge spanning the width of a river |
| Landmark |  | Lower Manhattan skyline - June 2017 | a large body of water with a city in the background | high rise, city, urban, building, architecture | city skyline under blue sky during daytime | a large body of water with a clock tower on top of it |
| Landmark |  | the Eiffel Tower seen from the Tour Saint-Jacques, Paris August 2014 | a large body of water with a city in the background | building, spire, architecture, tower, metropolis, city, urban | eiffel tower in paris france during daytime | a large clock tower towering over a city |
| Landmark |  | Colosseum in Rome, Italy - April 2007 | a castle on top of Colosseum | building, architecture, person, castle | brown concrete building under blue sky during daytime | a tall building with a clock on it |
| Landmark |  | San Paulo Catedral | a group of people walking in front of a large church with São Paulo Cathedral in the background | cathedral, building, architecture, church, tower, spire, person, downtown, city, urban" | white concrete building near green trees during daytime | a large building with a clock tower on top |
| Landmark |  | Skyscrapers of Shinjuku 2009 January | a large body of water with a city in the background | city, urban, building, high rise, downtown | city skyline under blue sky during daytime | a view of a city with a clock tower |
| Landmark |  | Lady Liberty under a blue sky | a tall clock tower sitting next to a statue with Statue of Liberty in the background | sculpture, art, statue, person | statue of liberty new york | a statue of a man on a horse |
| Landmark |  | Castel Sant'Angelo | a group of people standing in front of a building | architecture, building, person | brown concrete building with statue of man and woman | a large building with a clock on it |
| Landmark |  | Bellas Artes | a large building with Palacio de Bellas Artes in the background | building, architecture, person, mansion, housing, house, palace | people walking near beige concrete building under blue sky during daytime | a tall building with a clock on it |
| General |  | Sears House 115 | a house that has a sign on the side of a building | plot, diagram, plan | white and brown house illustration | a black and white photo of a old building |
| General |  | wedding and engagement Rings | a close up of a hand | person, accessories, ring, jewelry | silver diamond ring on persons finger | a person holding a pair of scissors in their hands |
| General |  | "teens sharing a song the little story" | a close up of a man and a woman looking at the camera | person, face | woman in blue crew neck shirt wearing white earbuds | a couple of women standing next to each other |
| General |  | outdoors man portrait | a man wearing a striped shirt | person, man | man in brown and white plaid button up shirt | a man in a suit and tie sitting down |
| General |  | woman mechanic working on engine | a young boy wearing a hat | helmet, clothing, person, hardhat, machine | woman in black and red plaid shirt wearing yellow hard hat | a man in a hat is sitting on a motorcycle |
| General |  | wheat close-up | a bird sitting on top of a grass covered field | plant, produce, food, grain, vegetable, bird, vegetation | brown wheat field during daytime | a bird sitting on top of a tree branch |
| General |  | good food display | a close up of many different types of food | plant, food | red and yellow chili peppers and garlic | a table topped with lots of different types of food |
| General |  | loaves of bread in a basket | a close up of food | bread, food, bun | brown bread on blue and white checkered textile | a box filled with different types of donuts |
| General |  | cowgirl creamery point reyes | a half eaten doughnut | bread, food, brie | sliced of bread on black textile | a close up of a plate of food |
| General |  | complete brown sugar rapadura | a close up of a teddy bear | sugar, food | brown powder on white ceramic plate | a close up of a cake on a plate |
| General |  | a small cup of coffee | a cup of coffee on a table | spoon, coffee cup, cup, saucer, pottery | white ceramic cup with black liquid | a coffee cup sitting on top of a table |
| General |  | longjing tea steeping in gaiwan | a bowl of food on a plate | pottery, jar, vase, plant, beverage, green tea | green leaves on white ceramic bowl | a bowl of soup with a spoon in it |
| General |  | Parigi - Tutankhamun, the Pharaoh's Treasure - miniature ivory game board | a camera on a table | toy, old, vintage, history | white and brown wooden lego blocks | a remote control sitting on top of a table |
| General |  | youth soccer indiana | a group of young men playing a game of football | person, people, team, soccer, ball, sport, team sport, soccer ball, football | 3 boys playing soccer on green grass field during daytime | a group of young children playing a game of soccer |
| General |  | football 1996 | a group of young men playing a game of football | person, people, soccer, team sport, team, soccer ball, sport, ball, football | man in red and white soccer jersey kicking soccer ball on soccer field during daytime | a group of young men playing a game of soccer |
| General |  | glass of milk | a glass of milk next to a cup of water | beverage, milk | white liquid in clear drinking glass | a cup of coffee sitting next to a plate of food |
| General |  | times square night 2013 | a group of people walking on a city street | urban, downtown, city, building, metropolis, person, architecture, pedestrian | people walking on street during night time | a crowd of people walking down a street |
| General |  | drinking water | a close up of a glass vase | outdoors, indoors, nature | water drop in close up photography | a glass vase with some flowers in it |
| General |  | common alcoholic beverages | a bottle of wine on a table | alcohol, beverage, beer, bottle, beer bottle | three brown glass bottles on brown wooden table | a bottle of wine sitting on top of a counter |
| General |  | market vegetables | a bunch of different vegetables on display | shop, grocery store, plant, market | green cucumber on fruit stand | a produce stand with various vegetables and fruits |
