Climate change and insect declines were worrying ecologists long ago - 1850-1900

 

Since various studies have suggested that insect biomass has declined about 75% in recent decades, one could say that there were four times as many insects. Large biomass losses also occurred in the time earlier (agricultural revolution, as e.g. described in Rachel Carlson's book 'Silent Spring'). The 75% loss refers to the latest decades only. How many insects were there around 1900 or before? Why is there so little data?

Klimaerwärmung und Insektensterben waren lange her bereits besorgniserregend für Ökologen - 1850-1900

 

Seit verschiedene Studien nahegelegt haben das in den letzten Jahrzehnten 75% der Biomasse der Insekten zurückgegangen ist, kann man im Umkehrschluss sagen das es davor vier mal so viele Insekten gab. Große Verluste gab es in der Zeit davor auch schon (Agrarrevolution, wie z.B. beschrieben in Rachel Carlsons Buch 'Der stumme Frühling'). Der Verlust von 75% bezieht sich auf die letzte Phase. Wie viele Insekten gab es um 1900 oder davor? Warum gibt es so wenig Daten?

Species lists from different sources - how to synchronise names with the GBIF database in R

Species lists from different sources:
how to synchronize names with the GBIF database in R,
how to compile a list of Syrphidae from Switzerland and
how to determine which species are lacking in common identification keys



You may know this problem: you have 2 or more species list from different sources and they use different synonyms for the same species. You want to merge the lists to a single one that has only unique species. In this post you find:

1. Getting species lists from books in to R
2. Using R to compare these species list with the GBIF species database and merge them (including 'tricks', e.g. to extract names from a multidimensional list)
3. Discussion of common problems to take in to account
4. The final list of Syrphidae nicely formatted for printing

Case study description
The case study concerns European Hoverflies (Syrphidae). Unlike plants or mammals, this insect group has been subject to many synonymizations which results in very dissimilar name lists over  time frames of a decade. I want to identify Syrphidae species from the Alps (mainly Switzerland), with two commonly used identification keys that do not include all alpine species. I need to know which species have to be identified with additional keys and which genera are safe to identify with the common keys. I took the species list of Switzerland from the website World of Syrphidae. For convenience and usability I also downloaded the German species list.

For Northwest Europe there are two recent major identification works:
The first covers (almost) all lowland species:
Van Veen, M. (2004) Hoverflies of Northwest Europe: identification keys to the Syrphidae. 256pp.
Van Veen published an addendum to his book on his website which enables identification of even more Syrphidae. I copied those species names manually into a txt file, an this list will be used as well.
The second work comes in two volumes and covers many alpine species:
Bartsch, H. et al. (2009) Blomflugor: Syrphinae. Nationalnyckeln till Sveriges flora och flora, DH53a. Artdatabanken, SLU, Uppsala. 406 pp.
Bartsch, H. et al. (2009) Blomflugor: Eristalinae & Microdontinae. Nationalnyckeln till Sveriges flora och flora, DH 53b. Artdatabanken, SLU, Uppsala. 478pp.

1. Getting the species lists from books and web pages into R

When you get experienced with this, it is faster than typing every single species manually.

• Scan the species list from your book, either with a modern copy machine or flatbed scanner (A digital camera often yield too strong distorted images that do not give a good text recognition.)
• Get a good text recognition (OCR, optical character recognition). I prefer to do a good OCR with Acrobat pro on a pdf made from the scans. There is also free OCR software that can do a good job on tiff and bmp images (here some software suggestions to try).
• For a good result, make sure that you have one column with species on your input. If you have e.g. 2 columns on a single page or printed  2 pages on a single pdf page, you have to edit the images a bit. For a pdf you can do the following. Crop your pdf to the left side first and export as tiff image. Rename from filename.* to filename_A.* (you can do this with e.g. total commander under Files -> multi-rename tool). Crop your pdf to right side and export as tiff images. Rename from filename.* to filename_B.* Now combine files to pdf again. You now should have a single list of species per page.
• Perform the OCR. Either export to text (save as other, txt (accessible), or select parts of the list manually and copy it into a *.txt file.
• Clean up the file. Keep a list of what to "find and replace": e.g. / must be l. 
• Import the text file to excel or libre office calc. The delimiter is space. First 2 columns are genus and species. The rest may be rubbish or things like authority or common name and may be deleted. If you like to keep any of this (for example authority) you can do the following. Select all remaining columns (except genus/species), paste into a txt file, put behind the authority a sign for separation, instead of a space. In my example I had the kind of format: "Authority - Name". I replaced " - " by "-" to get a nice separation without spaces. Import and choose delimiter like "-". 

Everything should now be in a perfect table! Scan through for further OCR errors, remaining ones can be identified in R. For use in R, I named the columns "GenSpec" for the species name, "FullGenSpec" for the species with authority, and "Gen" and "Spec" for using both elements separately.

2. Using R to compare species list with the GBIF database

Start and read the files with species lists:

################################_
### Load packages and functions, set working directory     ####
################################_

setwd("C://.../specieslistsR")

#install.packages("taxize")
#install.packages("dplyr")

require(taxize) #for connecting to gbif species lists
require(dplyr) #for bind_rows

# a nice function for removing accidential spaces at end of species name
trimendspace <- function (x) sub("\\s+$", "", x) 


################################_
### Read files     ####
################################_


#DATAFILES

list_vV <- read.table ("VanVeen_species_list.txt",na.strings="NA",head=T,sep="\t", dec=",", encoding = "utf-8") #encoding is relevant when OCR on scanned documents was performed
head(list_vV)
for (i in c(1:5))    {list_vV[,i] <- trimendspace(as.character(list_vV[,i]))}

list_vVadd <- read.table ("vanVeen_Addendum.txt",na.strings="NA",head=T,sep="\t", dec=",", encoding = "utf-8")
head(list_vVadd)
for (i in c(1:1))    {list_vVadd[,i] <- trimendspace(as.character(list_vVadd[,i]))}

list_Ba <- read.table ("Bartsch_species_list.txt",na.strings="NA",head=T,sep="\t", dec=",", encoding = "utf-8")
head(list_Ba)
for (i in c(1:5))    {list_Ba[,i] <- trimendspace(as.character(list_Ba[,i]))}

list_CH <- read.table ("checklist_CH.txt",na.strings="NA",head=T,sep="\t", dec=",", encoding = "utf-8")
head(list_CH)
for (i in c(1:3))    {list_CH[,i] <- trimendspace(as.character(list_CH[,i]))}

list_DE <- read.table ("checklist_DE.txt",na.strings="NA",head=T,sep="\t", dec=",", encoding = "utf-8")
head(list_DE)
for (i in c(1:5))    {list_DE[,i] <- trimendspace(as.character(list_DE[,i]))}

Two functions - the heart of the code:

#function for consulting Gbif and putting the right information in a dataframe
#with the species list copied to 'Gbifask'

func.GbifQuery <<- function() 
{
  #get current status for a list of species from gbif
  listOfResults <<- get_gbifid_(Gbifask)
  
  cat("\n", file = "gbifunknown.txt", append = F) #File for species names without retuns; F: clean file
  
  #put data in dataframe
  for (i in c(1:length(specieslist)))
  {
    if(i==1) # first entry used to start new dataframe
    {
      #first entry is the exact match if any present
      #Gbifid <- listOfResults[[i]][1,]
      listOfResults[[i]][,"enteredname"] <<- names(listOfResults[i])
      Gbifid <<- listOfResults[[i]]
    }  else # second an further are added to dataframe by bindrows
    {
      if(length(listOfResults[[i]][])>0) # all valid entries
      {
        listOfResults[[i]][,"enteredname"] <<- names(listOfResults[i])
        Gbifid <<-  bind_rows(Gbifid, listOfResults[[i]])      
      } else # species unknown to gbif, write to file diectly
      {
        cat("\n", names(listOfResults[i]) ,file = "gbifunknown.txt", append = T)
      }
    }
  }
}


#function for corrections with synonym lists

func.correctlists <<- function()   {
  speciescorrect <<- 0
  
  for (i in c(1:nrow(CorrectionList)))     #go through list with synonyms
  {
    for (Listname in ToCorrect)           #go through series of lists to correct
    {
      Specieslist <<- get(paste(Listname))
      
      speciescorrect <<- as.character(CorrectionList[i, "OldGenSpec"])
      if(length(Specieslist[Specieslist$GenSpec==speciescorrect, "GenSpec"])>0)
      {
        #first save old name
        if(correctiontype=="Synonym")
        {Specieslist[Specieslist$GenSpec==speciescorrect, "SynSpec"] <<- as.character(CorrectionList[i, "OldGenSpec"])}
        #than replace
        Specieslist[Specieslist$GenSpec==speciescorrect, "GenSpec"] <<- as.character(CorrectionList[i, "NewGenSpec"])
      }
      assign(paste(Listname), Specieslist, inherits = TRUE)
    }
  }
}

Merge the species lists:

#Merge the species lists
speciesbind <- bind_rows(list_vV, list_vVadd, list_Ba, list_CH, list_DE)
speciesbind <- speciesbind[order(speciesbind$GenSpec),]

#remove duplicates

#which full species names (with authority) are duplicated?
speciesauthoritiesduplicated <- speciesbind[duplicated(speciesbind$FullGenSpec),"FullGenSpec"]
#remove them
set1 <- speciesbind[-which(speciesbind$FullGenSpec %in% speciesauthoritiesduplicated),]

#check what remains:
speciesduplicated <- set1[duplicated(set1$GenSpec),"GenSpec"]
set1[which(set1$GenSpec %in% speciesduplicated),"FullGenSpec" ]

Look at the saved table. Slightly different spelled authority names can be dismissed, clearly different authority names need to be retained.

Consult the GBIF database with the raw species list:

#make the species list. In our case all duplicates were Authority spelling variations
specieslist <- unique(speciesbind$GenSpec)

#Now find the synonyms
Gbifask <- specieslist
func.GbifQuery() #returns Gbifid

First get a list of typos:

# which entry has no exact match?

# get list of entries which have an exact match
matchingnames <- Gbifid[Gbifid$matchtype=="EXACT",  ]
matchingnameslist <- matchingnames[,c("enteredname")]

#get a list of entries which have a fuzzy match
unresolvednames <- Gbifid[Gbifid$matchtype=="FUZZY",  ]  
#reduce the fuzzy list; filter out entries that also have an exact match
unresolvednames <- unresolvednames[-which(unresolvednames$enteredname %in% matchingnameslist), ]

#save this list, these are possible typos making them unrecognizable for GBIF
namestosave <- unresolvednames[,c("enteredname", "scientificname", "canonicalname","species","matchtype")]
write.table(namestosave, "possibletypos.txt", row.names = FALSE, col.names = TRUE, sep = "\t", dec=".") 

• Open the saved file in a spreadsheet program, with the goal to make make a correction list
• FUZZY: mostly misspellings (by OCR) or male/female spelling changes - make a list for corrections. 
• the column "entered name" gets "OldGenSpec" and the column "canonicalname" gets "NewGenSpec" (synonyms in "species" are handled later). This is a translation list to replace the typos.
• Carefully check all entries. You can also consult the list of valid species names to see if one of  the fuzzy ones matches. If the fuzzy match is unclear and you like to see if the Authority matches somewhere, you can e.g. use: speciesbind[grep("^Pipiza ju",speciesbind$GenSpec),"FullGenSpec"]
• Leave alone names that do not match at all! Remove them.
• Look for crosslinking changes in the list (new name also occurs in old name list): forbidden! (here not present)
• Name your file TypoCorrection.txt

Remove the typos:

### now replace typos in all lists with correction function ###

#load correction file
corrections <- read.table ("TypoCorrection.txt",na.strings="NA",head=T,sep="\t", dec=",")
head(corrections)
for (i in c(1:2))    {corrections[,i] <- trimendspace(as.character(corrections[,i]))}

ToCorrect <- c("list_vV","list_vVadd","list_Ba","list_CH","list_DE")
CorrectionList <- corrections
correctiontype <- "Typo"
func.correctlists() 

Get synonyms round 1

#Merge species lists
speciesbind <- bind_rows(list_vV, list_vVadd, list_Ba, list_CH, list_DE, list_SA)
speciesbind <- speciesbind[order(speciesbind$GenSpec),]
specieslist <- unique(speciesbind$GenSpec)

#get current status for species from Gbif and put in dataframe
Gbifask <- specieslist
func.GbifQuery() #returns Gbifid

### Identify synonyms and evaluate thoroughly ###

#having solved all Fuzzy returns without an accepted counterpart in the data in the previous step, we now only have fuzzy returns which can be ignored, because they have a recognized valid name as well
#the names remaining are species with exact match with either synonym false and synonym true

#in first instance you would try to get the synonyms in this way:
resolvenames <- Gbifid[Gbifid$matchtype=="EXACT" & Gbifid$synonym==TRUE,  ]  #list of names marked as synonym
namestosave1 <- resolvenames[,c("enteredname", "scientificname", "canonicalname","species")]
#however this also lists all other known synonyms that are not relevant here. 
#the list needs to be reduced to the names that were originally entered - and see which of those ar now considered synonyms
getnames <- namestosave1[, "enteredname"]  
resolvenames2 <- Gbifid[which(Gbifid$enteredname %in% getnames),  ]  #reduce Gbifid
resolvenames3 <- resolvenames2[resolvenames2$matchtype=="EXACT", ]  #retain those labelled as exact match
namestosave2 <- resolvenames3[,c("enteredname", "scientificname", "canonicalname","species", "synonym")]
write.table(namestosave2, "synonymsfromgbif.txt", row.names = FALSE, col.names = TRUE, sep = "\t", dec=".") # save this table

• open the saved file in a spreadsheet program, with the goal to make make a synonym list to replace 'old' names
• the column "entered name" gets "OldGenSpec" and the column "species" gets "NewGenSpec"
• when one name has both an entry for synonym true and false, choose the right one: you can check for options in your data like this: speciesbind[speciesbind$GenSpec=="Eristalis lineata","FullGenSpec"] If multiple sources (duplicates in speciesbind) consistently use the same authority, you can use that one (if synonym=false you can thus delete both entries). You can also check other sources for the right authority
• go through all remaining names and decide what is the right and accepted name
• at the end of this blog post I will give some hints on doing this, since GBIF is far from perfect when it comes to insect classification in general.
• for now accept the synonym list you made
• look for crosslinking changes in the list (new name also occurs in old name list): forbidden! (here not present)
• name your file Synonyms1.txt

Replace synonym names with most actual one:

# Prepare new columns and variables

synonyms1 <- read.table ("Synonyms1.txt",na.strings="NA",head=T,sep="\t", dec=",")
head(synonyms1)
for (i in c(1:2))    {synonyms1[,i] <- trimendspace(as.character(synonyms1[,i]))}

ToCorrect <- c("list_vV","list_vVadd","list_Ba","list_CH","list_DE")
CorrectionList <- synonyms1
correctiontype <- "Synonym"
func.correctlists() 

Resolve remaining issues:

#Merge species lists
speciesbind <- bind_rows(list_vV, list_vVadd, list_Ba, list_CH, list_DE, list_SA)
speciesbind <- speciesbind[order(speciesbind$GenSpec),]
specieslist <- unique(speciesbind$GenSpec)

#get current status for species from Gbif and put in dataframe
Gbifask <- specieslist
func.GbifQuery() #returns Gbifid

#get list of species names with exact match 
matchingnames <- Gbifid[Gbifid$matchtype=="EXACT",  ]
matchingnameslist <- matchingnames[,c("enteredname")]

#get unresolved names
unresolvednames <- Gbifid[Gbifid$matchtype!="EXACT",  ]  
#filter out those with exact match
unresolvednames <- unresolvednames[-which(unresolvednames$enteredname %in% matchingnameslist), ]
#save unresolved names to file
namestosave <- unresolvednames[,c("enteredname", "scientificname", "canonicalname","species","matchtype")]
write.table(namestosave, "gbifunclear.txt", row.names = FALSE, col.names = TRUE, sep = "\t", dec=".") 

# open gbifunknown.txt and gbifunclear.txt and edit as spreadsheet
# there are probably more misspellings and/or synonyms to resolve
# make a new file with synonyms, name it Synonyms2.txt

# replace synonyms

synonyms2 <- read.table ("Synonyms2.txt",na.strings="NA",head=T,sep="\t", dec=",")
head(synonyms2)
for (i in c(1:2))    {synonyms2[,i] <- trimendspace(as.character(synonyms2[,i]))}

ToCorrect <- c("list_vV","list_vVadd","list_Ba","list_CH","list_DE")
CorrectionList <- synonyms2
correctiontype <- "Synonym"
func.correctlists() 

Combining the different species lists in one table and solve multiple synonyms:

#challenge: some accepted species names have multiple synonyms that need to be merged

#Merge species lists
speciesbind <- bind_rows(list_vV, list_vVadd, list_Ba, list_CH, list_DE)
speciesbind <- speciesbind[order(speciesbind$GenSpec),]
speciesbind <- speciesbind[!is.na(speciesbind$GenSpec),]

#make unique species list
#specieslist <- unique(speciesbind[, c("GenSpec","SynSpec")]) #does not work: many duplicate entries
specieslist <- unique(speciesbind$GenSpec)

#deal with  multiple synonyms
speciessynonyms <-  unique(speciesbind[speciesbind$SynSpec!="0", c("GenSpec","SynSpec")  ])
specieslistwithdouble <- speciessynonyms[duplicated(speciessynonyms$GenSpec),c("GenSpec")]

#merge synonyms
specieswithdouble <- setNames(as.data.frame(specieslistwithdouble),c("GenSpec"))
specieswithdouble$DSynSpec <- as.character("0")
for(i in c(1:nrow(specieswithdouble)))
{
  allsynonyms <- paste(as.character(speciessynonyms[speciessynonyms$GenSpec==specieslistwithdouble[i] & !is.na(speciessynonyms$GenSpec),c("SynSpec")], sep=","))
  syncombi <- NULL
  for (j in c(1:length(allsynonyms)))
  {
    if(j==1)
      syncombi <- paste0(allsynonyms[j])
    else syncombi <- paste0(syncombi,", ", allsynonyms[j])
  }
  specieswithdouble[i,"DSynSpec"] <- syncombi
}

#replace single synonyms with double synonyms
for (i in c(1:nrow(specieswithdouble)))
{
  speciessynonyms[speciessynonyms$GenSpec==specieswithdouble[i, "GenSpec"] & !is.na(speciessynonyms$GenSpec), "SynSpec"] <- as.character(specieswithdouble[i, "DSynSpec"])  
} 
#reduce duplicates
speciessynonyms <- unique(speciessynonyms)
speciessynonyms <- speciessynonyms[!is.na(speciessynonyms$GenSpec) ,]

#merge species list with synonyms
speciestable <- merge(setNames(as.data.frame(specieslist),c("GenSpec")), speciessynonyms, by="GenSpec", all.x = TRUE)

Other steps: 

add genus name

get a clean standardized authority name for your species
get a column for year of description, which is helpful for judging older species list.

#add Genus name, work with string splitting 
#strsplit(specieslist, " ")
speciestable$Genus <- 0
for (i in c(1:nrow(speciestable)))
{
  speciestable[i,"Genus"] <- strsplit(as.character(speciestable$GenSpec), " ")[[i]][1]
}

#Get fresh Authority names
Gbifask <- specieslist
func.GbifQuery() #returns Gbifid

getnames <- Gbifid[(Gbifid$matchtype=="EXACT" & Gbifid$synonym==FALSE),c("enteredname", "scientificname") ]  
length(specieslist)-nrow(getnames) #some, because of one reversed name change

authority <- setNames(Gbifid[Gbifid$matchtype=="EXACT", c("enteredname", "scientificname") ] ,c("GenSpec", "ScientificName"))  

speciestable <- merge(speciestable, authority, by="GenSpec", all.x = TRUE, all.y = FALSE)

#authorities not available from Gbif, get from own input files
getAuthority <- speciestable[is.na(speciestable$ScientificName), "GenSpec"]

if(length(getAuthority)>0)   #if(nrow(getAuthority)>0)
{
  getAuthority <- setNames(as.data.frame(getAuthority), c("GenSpec"))
  # here Scaeva selenetica does not occur in the list, take the other name?
  getAuthority$ScientificName <- list_vV[list_vV$GenSpec==getAuthority$GenSpec, "FullGenSpec"]
  for (i in c(1:nrow(getAuthority)))
  {  genspec <- getAuthority[i,"GenSpec"]
  ScientificName <- getAuthority[i,"ScientificName"]
  speciestable[speciestable$GenSpec==genspec, "ScientificName"] <- ScientificName
  }
}

#check double authorities
speciestable[duplicated(speciestable$GenSpec),c("GenSpec","ScientificName")]
speciesauthoritiesduplicated <- speciestable[duplicated(speciestable$GenSpec),"GenSpec"]
speciestable[which(speciestable$GenSpec %in% speciesauthoritiesduplicated),]

#from the returned list, choose which to remove
listofnamestoremove <- c("Cheilosia griseiventris Vujic, 1994","Eristalis lineata Wahlberg, 1843",
                         "Eumerus argyropus Doleschall, 1857","Eumerus ruficornis Macquart, 1829",
                         "Merodon rufus (Macquart, 1835)", "Microdon analis Curran, 1940",
                         "Microdon myrmicae", "Paragus quadrifasciatus Say, 1824",
                         "Rhingia rostrata Scopoli, 1763", "Sphaerophoria interrupta Jones, 1917")

#remove them
speciestable <- speciestable[-which(speciestable$ScientificName %in% listofnamestoremove),]

#check if certain authorities were given to multiple species

speciestable[duplicated(speciestable$ScientificName),c("GenSpec","ScientificName")]

#cleanup: remove one non-meaningfull entry:
speciestable <- speciestable[speciestable$GenSpec!="Cheilosia melanura ssp. Rubra",]

# add year of description
speciestable$ID_Year <- 0
for (i in c(1:nrow(speciestable)))
{
  speciestable[i,"ID_Year"] <- substr(strsplit(as.character(speciestable$ScientificName), ", ")[[i]][2], 1, 4)
}

Compare the species lists and put the information in new columns and write the final table

# now see which species is in which list

#KEY van Veen
speciestable$Key_vVeen <- ifelse(speciestable$GenSpec %in% list_vV$GenSpec, "y", "n")
speciestable$Key_vVeen <- ifelse(speciestable$GenSpec %in% list_vVadd$GenSpec, "A", speciestable$Key_vVeen)

#Genus completely covered? #Make table with species per genus and 'no' counts
vVeensum <- setNames(as.data.frame(table(speciestable[speciestable$Key_vVeen=="n","Genus"])), c("Genus", "NumOfNo"))
speciestable$Key_vVeenGen <- ifelse(speciestable$Genus %in% vVeensum$Genus, "n", "y")

#KEY Bartsch
speciestable$Key_Bartsch <- ifelse(speciestable$GenSpec %in% list_Ba$GenSpec, "y", "n")
#Genus completely covered? #Make table with #species per genus and 'no' counts
Basum <- setNames(as.data.frame(table(speciestable[speciestable$Key_Bartsch=="n","Genus"])), c("Genus", "NumOfNo"))
speciestable$Key_BartschGen <- ifelse(speciestable$Genus %in% Basum$Genus, "n", "y")

#Checklists CH, DE, S-A
#Switzerland
speciestable$PA_CH <- ifelse(speciestable$GenSpec %in% list_CH$GenSpec, "p", "a")
speciestable$KeyCH <- ifelse( (speciestable$Key_vVeen=="n" & speciestable$Key_Bartsch=="n" & speciestable$PA_CH=="p"), "n", "y")
speciestable$KeyCH <- ifelse( (speciestable$PA_CH=="a"), "-", speciestable$KeyCH)
#Germany
speciestable$PA_DE <- ifelse(speciestable$GenSpec %in% list_DE$GenSpec, "p", "a")
speciestable$KeyDE <- ifelse( (speciestable$Key_vVeen=="n" & speciestable$Key_Bartsch=="n" & speciestable$PA_DE=="p"), "n", "y")
speciestable$KeyDE <- ifelse( (speciestable$PA_DE=="a"), "-", speciestable$KeyDE)
#small optic correction: 
speciestable$SynSpec <- as.character(speciestable$SynSpec)
speciestable$SynSpec <- ifelse(is.na(speciestable$SynSpec), "-", speciestable$SynSpec)

speciestable <- speciestable[ , c("GenSpec", "Genus", "SynSpec", "Key_vVeen", "Key_vVeenGen","Key_Bartsch", "Key_BartschGen", "PA_CH","KeyCH", "PA_DE", "KeyDE") ]
# "ScientificName","ID_Year" can be added in subsequent step

#FIN
write.table(speciestable, "SyrphidSpecies&Keys&Presence.txt", row.names = FALSE, col.names = TRUE, sep = "\t", dec=".") 

The full R script and its input files can be downloaded here.


3. Discussion of common problems to take in to account

Unfortunately the GBIF database is often wrong about the names of Syrphidae, like:

• Synonyms are very wrong (e.g. Chrysotoxum festivum and Xanthogramma festiva are separate species, not synonyms)
• Old and new name should be reveresed (many examples, e.g. Brachypalpoides lentus and Xylota lenta)
• Genus name changes should be consistent, but aren't always (resulting in two genera names instead of one, e.g. Anasimyia vs Lejops -> choose Anasimya and reverse respective synonyms)
• Synonymization merging species that should remain separate (e.g. Eristalis picea and    Eristalis rupium are separate species, not synonyms)
• Some species do not have an exact GBIF match, but are valid anyway (e.g. Eristalis similis, Myathropa florea and Sphaerophoria interrupta)
• The "Syrph the Net (StN) Species Accounts" is the most elaborate documentation about European Syrphid species, synonyms and doubtfull species names. I trusted this source more than GBIF and consulted it many times to get to my final list of species. 
• The Genus Cheilosia (not well handled by Gbif) is covered by a separate publication: Claußen, C.; Speight, M.C.D. (2007): Names of uncertain application and some previously unpublished synonyms, in the European Cheilosia fauna (Diptera, Syrphidae). - Volucella 8, 73-86. Stuttgart.

As a final note: I had a very strange error caused by the OCR software. In some species names it turned the letter 'l' into a special character, that I could not convert to a 'l' in R in any way, leading to the strange fact that R handled e.g. 'Episyrphus balteatus' and 'Episyrphus balteatus' as being equal, but both unique at the same time. The only working solution was to re-type the specific names in the spreadscheet files manually, to get rid of these type of duplicates in the list.

The supplied synonym files used in my R-script include all those changes needed for the list I received from GBIF. Note that GBIF is constantly changing, possibly requiring a slightly different synonyization list.

Code for some of the manual work

# track more misspellings
similarspecies <- as.data.frame(specieslist)
similarspecies$specieslist <- as.character(similarspecies$specieslist)
similarspecies$short <-  substr(similarspecies$specieslist, 1, nchar(similarspecies$specieslist)-5)
templist <- similarspecies[duplicated(similarspecies$short), "short"]
similarspecies[similarspecies$short %in% templist,]

# check for double species names accepted under different genera
# similarnames <- as.data.frame(speciesbind[, c("GenSpec", "Genus", "Species")])
similarnames <- as.data.frame(specieslist)
# add Genus name and species name, work with string splitting 
similarnames$Genus <- 0
similarnames$Species <- 0
for (i in c(1:nrow(similarnames)))
{
  similarnames[i,"Genus"] <- strsplit(as.character(similarnames$specieslist), " ")[[i]][1]
  similarnames[i,"Species"] <- strsplit(as.character(similarnames$specieslist), " ")[[i]][2]
}
similarnames$short <-  substr(similarnames$Species, 1, nchar(similarnames$Species)-2)
doublenames <- similarnames[duplicated(similarnames$short), "short"]
doublenamestable <- similarnames[similarnames$short %in% doublenames,]
doublenamestable[order(doublenamestable$short),]

# I also recommend to do new Gbif Queries

# You can use an extra file for replacement "Extra synonyms.txt" 

# Or do manual replacements like this:

speciestable [speciestable $GenSpec=="Merodon subfasciatus", "GenSpec"] <- "Merodon cinereus" 

# "Doubtfull species" can be removed like this 

list_Chei <- read.table ("Cheilosia.txt",na.strings="NA",head=T,sep="\t", dec=",")
head(list_Chei)
for (i in c(1:4))    {list_Chei[,i] <- trimendspace(as.character(list_Chei[,i]))}
# Or remove species manually

speciestable <- speciestable[speciestable$GenSpec!="Eumerus emarginatus",]

4. The final list

The final formatted list for downloading

The final table has for both identification keys a colomn with yes/no for each species. The species supplied in the addendum by van Veen, are specified with "A". A genus completely covered by the key is marked with a vertical black line. For Switserland and Germany there is present/absent column, with an additional column whether or not the present species can be identified with one of both keys, with "n" highlighted. These are the species that require additional keys. Any species in the same genus identified with those two main keys need to be treated carefully, since it can also be one of the rarer species for which you need an additional key. Have fun identifying!
 

A poster with common wild bees in gardens

In my office I have a wonderful poster of common bees in dutch gardens, with excellent illustrations by Jeroen de Rond, distributed by Naturalis and EIS for free (link). Unfortunately, the poster has only dutch bee names, though I am more familiar with the scientific names. Also, my English- and German-speaking colleagues would love to know the names in their language! Hence, I put myself to the task to make a translation, resulting in this table, that you can put next to the poster to have all the different names. The common names are retrieved from "Field Guide to the Bees of Great Britain and Ireland" by Falk (2015) and "Taschenlexikon der Wildbienen Mitteleuropas" by Scheuchl & Willner (2016).

Extending to this, EIS provides a low quality A4 pdf of this poster (link). I thought it would be nice to add the scientific names to the Dutch version, so people can learn them. And, fortunately, it was a small effort to replace the dutch names for English or German names. So here is the result (with credit to EIS and Naturalis as original producers of this Poster):

Bees of Gardens - Dutch
Bees of Gardens - English
Bees of Gardens - German

Enjoy!

The search for the European Stalk-eyed Fly

Stalk-eyed Fly, Achias sp., male (Australia)

Stalk-eyed flies (Diopsidae) are fascinating insects. Males have their eyes on long stalks and the longer the stalks, the more attractive they are for females. It is quite unpractical to have such 'long eyes' and it is thought that when you can deal with such unpractical eyes, you must be quite healthy and therefore you are attractive when you have longer eyes (handicap principle).
When I heard that an European member of this fly family, Sphyracephala europaea, lives in Hungary and we had plans to visit Hungary on a future holiday anyway, I decided to go and search for it.

There is not much know about this species, because only few people had the luck to find one in the 20 years after it's discovery in 1996. The original article (Papp 1997) and the key to the European fly families by Oosterbroek, as well as several blogs reporting about a lucky find (here and here), do give some basic information.


Sphyracephala europaea lives along waters at shadow-rich places often occurring in large numbers together. These small flies (4 mm) seem to prefer the direct shore of the river. Flies were found from September to November, but also in June. The pictures in the blogs give an additional feel of the habitat of the fly.

We stayed a few days in Mako, south east of Szeged, so that I could search along the river Mures, which s known as one of the places where this fly can be found.

The fluvial forest that I could see from the campsite looks very promising for finding S. europaea or some other rare species. The challenge is to find some open spots, where I can reach the river bank and optimally find some flat shore to search.

The first day after arrival, we decided to rent a bike and get us a decent map of the environment. It is generally the best option to get a local map with detailed bike and hike tracks; the most recent maps are normally sold locally. The alternative with google maps, was not an option anymore, since the internet connection at the campsite was dead. The bike rental worked out well, at a hotel nearby. Arriving at the tourist office, people were very happy with us. Mako has been made more accessible for tourists the past years, with an official tourist office, but apparently hardly tourists. They tried to sell us postcards of Mako. Unfortunately they did not have a detailed map, only road maps just as detailed we already had, not sufficient to identify good places to look for S. europae ... The local bookstore had more maps but not more detailed, and a large pile of the, apparently single type, postcard of Mako.

We set of for a bike trip along the north side of the river. It was an interesting trip over the river dike and enjoying the countryside. Only occasionally there were places where I could reach the shore. Mostly it was without a sandy shore, with the land abruptly ending with a meter downward into the water. It was hard to look for insects here. And as it may not surprise, I was not successful.

The second day, I decided to try the other (south) side of the river. I found a place with a small path only a few meters from the shore, probably for people going fishing.

A place being ecologically very interesting, with many old trees, occasionally flooded and with dense vegetation.


This fluvial forest 'looks like a jungle'.

Soon, I started to see flies. Flies of all sorts...

This is a Black scavenger fly (Sepsidae)


A Stiletto fly (Therevidae)





This could be a Sapromyza sp. (Lauxaniidae) or a Marsh fly (Sciomyzidae)



Probably a Snipe fly (Rhagionidae)



Very hard to tell, a Lance fly (Lonchaeidae)?, a Scuttle fly (Citrago sp., Phoridae)? or Cacoxenus indagator (Drosophildae)?


A Long-legged fly (Dolichopodidae) or a Stretched-foot fly (Tanypezidae)?



These thick long antenna are often indicating that these are Thick-headed flies (Conopidae), but probably these are 'Parasite flies' such as Cylindromyia bicolor (Tachinidae), indicated by the stiff hairs on the body and the red coloring.



A Robber fly (Asilidae), easily recognizable by the deep valley between the eyes.










With >5 cm too large to be a fly. It is an adult of another rarely seen insect group, Antlions (Myrmeleontidae), belonging to the insect order of net-winged insects (Neuroptera); adults are occasionally seen close to river shores, while funnels made by larvae are more commonly seen.



A Scorpion fly (Panorpidae). Not a fly (Diptera) at all, but belonging to the insect order Mecoptera.
Long antenna, long snout and a scorpion tail!

Unfortunately still no sign of a stalk-eyed fly ...


Near the campsite there was a small patch of sandy shore which may be the better habitat.


























The first insect I found was a Pygmy mole cricket (Tridactylidae).




This is Xya variegata, known from the Mures river banks. Quite nice!



 

And what is this?


This may be a shore bug, Saldula sp.




Finally, a small fly!



closer... This patterning is typical for the Genus Anthomyia in the family Root-maggot flies (Anthomyiidae).


Another small fly!

Is this the rare Leiomyza dudai the family Asteiidae, as Walter reports in his blog? Or maybe it is ....


Unfortunately, this day I still found no sign of a stalk eyed fly.


On the third day, we had to leave for our home trip. On the way back, we decided to stop near the river bank of the Mures river, but it was difficult to see on our road map where to stop the car. Our attempt to get close, resulted in a forest hike, that ended 50-100 metres from the river bank, with too dense shrubs to get trough. It was the end of the search.

Alleviating the disappointment, I found this beautiful butterfly:

The common glider - Neptis sappho




And finally, I also could make an acceptable photo of a dragonfly in flight!


I left for home, not with empty hands, but with a sight of the stalk-eyed fly neither. It was a pity, but I did my best. Home again, I finally could do a decent google map investigation with a lot of zooming. I found many additional places that would have been suitable to search! That was a bit depressing conclusion, but I never would have prepared on forehand a map so detailed as would haven been required for this... And to be honest, with probably the main flight period in September to November, I may had been unsuccessful anyway in finding this fly. It was an interesting quest for a fly with many interesting discoveries on the way!
(The fly at the startz

Convenient life-hack for an entomologist - solution to troublesome pin holding

If you ever have tried to identify a pinned insect, you may have been either irritated that holding the pin in your hand to see all sides is exhausting your muscles, or have been irritated that the pin is fixed somewhere and you can't see the side you need to observe. This very simple aid solves the problem.

Take a rectangular piece of polyethylene foam used as bottom inlay for insect boxes (or use something similar), about 5 by 5 cm. Take an apple corer and rotate very slowly with mild pressure into the foam to get the nice round hole in it. It exactly fits a finger! I usually use it on my forefinger or my thumb; sometimes I use two at the same time! And you can pin multiple specimens on one at the time and compare them. A very convenient life-hack for an entomologist.