The impact of cultural diversity on firm innovation: evidence from Dutch micro-data
© Ozgen et al.; licensee Springer. 2013
Received: 2 April 2013
Accepted: 1 September 2013
Published: 20 September 2013
An important question for firms and policy makers is whether the recruitment of foreign workers can boost innovation. Migration studies have demonstrated positive economic impacts of cultural diversity on productivity and innovation at the regional level, but the impacts at firm level are less well known. Merging data from four different sources, provided by Statistics Netherlands, we construct and analyze a unique linked employer-employee micro dataset of 4582 firms that includes qualitative information on firm innovation. We consider both the number of immigrants these firms employ and their cultural diversity. Potential endogeneity of migrant employment is addressed by an instrumental variables approach that accounts for the past geographic distribution of immigrants and the past culinary diversity of the municipality the firm is located in. We find robust evidence that firms employing relatively more migrants are less innovative. However, there is evidence of integration in that this effect is generally less strong or even absent for second generation immigrants. Moreover, firms employing a more diverse foreign workforce are more innovative, particularly in terms of product innovations. The benefits of diversity for innovation are more apparent in sectors employing relatively more skilled immigrants.
D22, F22, O31
KeywordsImmigration Innovation Cultural diversity Knowledge spillovers Netherlands
One of the major mechanisms for the diffusion of knowledge is the mobility of people. The geographic mobility of labor relocates human capital and its embodied knowledge and personal experiences (Döring and Schnellenbach, 2006). The importance of this knowledge transfer is increasing. The global economy is creating an unprecedented demand for a diversified and open-minded workforce while highly-skilled workers are seeking opportunities to utilize their human capital abroad and increase their income and experience. For example, an estimated 900,000 highly-skilled professionals entered the US between 1990 and 2000. Moreover, temporary workers account for one-sixth of the total IT workforce in the US (OECD, 2002). Such phenomena increase the rate of circulation of talent over space and across firms, leading to much greater diversity of the workforce than a few decades ago. Large, and often export-oriented, companies are seeking nowadays knowledge workers from all over the world (Saxenian, 2006; Page, 2007). For example, international transferees of multinational firms transmit knowledge in the form of experience and work practices across borders. It is an important question for firms and for governments to ask whether there are productivity-enhancing impacts from growing diversity among employees within firms.
A recent branch of migration literature has been focusing on the association between innovation and the presence of foreign workers. This literature, reviewed in, e.g., Ozgen et al. (2012), has tended to treat immigrants as a rather homogeneous group of employees. Potential skill complementarities and ethnic or cultural backgrounds of employees have often not been explicitly taken into account. Most studies use various firm characteristics as the main determinants of innovation and estimate a so-called knowledge production function (e.g., Audretsch and Feldman, 2004; Cohen and Levinthal, 1990). Such studies have often overlooked the characteristics of individual employees. The latter are clearly needed to assess the impact of employee diversity on the innovativeness of firms. To date – with the exception of Parotta et al. (2011), Lee and Nathan (2010), Simonen and McCann (2008) and Almeida and Kogut (1999) – there has been very little empirical evidence that takes the presence and characteristics of foreign employees into account in identifying the determinants of innovation at the firm level. We therefore focus in this paper on the effects of foreign employees with diverse backgrounds on firm innovation.
We utilize high-quality linked employee-employer data at the firm level, obtained from four different collections provided, in a secure environment and under a confidentiality agreement, by the Central Bureau of Statistics for the Netherlands (hereafter Statistics Netherlands). We combine survey and administrative information that relates to the period 2000-2002. We study by means of the resulting unique micro-dataset of 4582 firms whether the presence and relative numerical importance of migrants among the firms’ employees influences the firm’s self-reported innovativeness. We also test whether cultural diversity among these migrants is more conducive to innovation. Clearly, cultural diversity is a multidimensional concept (Wimmer, 2008), influenced by many factors (e.g. language, ethnicity, religion, identity, etc.). Due to data restrictions, we proxy cultural diversity among employees simply by birthplace but allocate birthplaces to culturally distinct groups. While this approach never fully represents cultural diversity, it has the advantage that birthplace information is objective and time invariant. The benefits for innovation obtained from a culturally diverse workforce are expected to be larger in sectors that employ high-skilled migrants and we account for this by excluding in some regressions sub-sectors that employ predominantly unskilled migrants.
Our study is the first to analyze innovation effects of foreign employees by means of a representative micro dataset at the firm level in the Netherlands. A potentially important issue is that of reverse causality. Foreign workers are not randomly assigned to more or less innovative firms. We address this issue by an instrumental variables (IV) approach that exploits the historical distribution of immigrants and past culinary diversity of the community the firm is located in. We exclude the hospitality sector in IV regressions of innovation because in that sector – in which ethnic restaurants employ migrants of the same or similar cultural background – the instruments would be correlated with the error term of the regression. We find that the instruments are adequate for the other sectors, with the overidentifying restrictions satisfied.
We proceed as follows. The next section briefly reviews a range of channels through which the employment of immigrants can impact on firm innovation. Section 3 then describes the strategy we adopt to identify the net effect of the presence and cultural diversity of immigrant employees on the responses firms give in the Netherlands Community Innovation Survey. The data are outlined and summarized in section 4. Section 5 reports the results of regression modeling and a range of feasible robustness checks, while section 6 sums up.
2. Theoretical linkages between immigration and innovation
An innovation is primarily the introduction of something radically new in the operations of a firm, obtained by means of analytical knowledge. The improvement of an existing product or the modification of an existing process or organizational arrangement can also be viewed as an innovation. Technological advances come from things that people do (Romer, 1990). Many worker characteristics, such as age, education, occupation, cultural background and language may affect knowledge acquisition and worker mobility (Poot, 2008). Current knowledge is the outcome of accumulated efforts. Each inventor begins from where its predecessors left off. The inventor explores the latest generation of products and services, and makes use of market knowledge that embodies a cumulative investment in time to develop products and processes (Grossman and Helpman, 1994). The presence of foreigners with diverse backgrounds in a labor market may serve to enrich this cumulative effort.
There have been many studies that have analyzed the impact of infrastructural and organizational aspects of firms on innovativeness. The importance of ideas rather than physical assets has only recently entered the innovation research agenda (Jones and Romer, 2010). The biggest change in the recent scientific literature is that it is now not the firm but the employees that are seen as a major source of innovation. One key focus of this new approach is the impact of foreign workers on the innovativeness and productivity of host firms and countries. Thus, one branch of this literature analyses the impact of foreign entrepreneurs, students and inventors on innovation (Stuen et al. 2012; Kerr, 2010; Kerr and Lincoln, 2010; Hunt and Gauthier-Loiselle, 2008; Lobo and Strumsky, 2008; Zucker and Darby, 2007; and Faggian and McCann, 2006). Evidence of spillover benefits from skilled foreigners joining an organization applies even to professional sports (see Alvarez et al. 2011). Another branch of this literature discusses the innovative and productive effects of externalities created by clusters of immigrant groups with diverse cultural backgrounds in particular regions (Ozgen et al. 2012; Niebuhr, 2010; Mazzolari and Neumark, 2009; Südekum et al. 2009). A major focus of this type of study is the average effect of immigrant diversity on regional productivity or innovation.
where the dependent variable I is the degree of innovative activity; the RD variable denotes an index of all kinds of R&D inputs; and HK represents an index of human capital inputs. The subscript i refers to the unit of observation, which is usually a firm or an establishment, and the parameters are estimated by log-linear regression.
However, there is a spatial dimension to innovation. This has led researchers to focus on the external forces and internal features of firms that stimulate innovation. Numerous studies scrutinized the significance of the external environment of a firm in terms of demand-supply links, industrial clusters, and diversity of production. The studies on the internal features of a firm emphasize the importance of a firm’s resources for innovation, such as R&D expenditures and the presence of high-skilled workers. Moreover, the ‘absorptive capacity’ of a firm determines whether locally produced knowledge will be utilized, improved and turned into creative outputs (Cohen and Levinthal, 1990; Caragliu and Nijkamp, 2012). This absorptive capacity may depend on the diversity of firm employment. Studies of inventors and their networks highlight the significance of spatial proximity and knowledge exchange among diverse groups of inventors (e.g. Agrawal et al. 2008). However, very few studies undertake their analysis at the establishment level, the smallest local production unit where the transfer of tacit knowledge is most likely to take place. Establishments can import new knowledge via employing ‘talent’ that already embodies such knowledge. Some firms are more likely to hire foreign workers than others, for example because they produce a wide range of products and services or because they sell to a wide range of countries (multinationals like Google are a perfect example). Alternatively, spatial proximity of talent at the firm’s location may also provide a critical mechanism for knowledge flows.
The Impacts of Immigration on Innovation: a Classification of Channels of Influence
• Positive self-selection of immigrants: e.g., intelligence, creativity, willingness to take risks, entrepreneurship, “star” knowledge workers (e.g. trained in host country universities)
• Youthfulness of immigrants: increased mobility, creativity, progressivity
• Cultural diversity among immigrants: knowledge spillovers, new ideas and practices, trade facilitation (networks, trust, institutional knowledge)
• Resilience of immigrants: enhances decision making
• Immigrant supply enables firm expansion: reduces shortages/vacancies of key personnel
• Fractionalization of employees: cultural and language differences and barriers, leading to communication problems, less trust, greater potential for conflict among staff, discrimination
• Greater labor intensity of production: lower reservation wages of immigrant workers lead to lower wage costs and, hence, lower capital investment in the short run (substitution effect), possibly offset by firm expansion in the long-run (output effect)
• Cultural diversity as an amenity: increased availability of ethnic goods and services in the community
• Population growth: agglomeration advantages, greater demand and gross fixed capital formation, with new technology embodied in new capital
• Community cohesion: bridging-type social capital leads to cross-cultural cooperation
• Sorting: Residential and labor mobility leads to greater spatial segregation: less cross-cultural relations and trade, lower spatial mobility and knowledge transfers
• Polarization: Bonding-type social capital leads to between-group conflicts
• Representation: Political fragmentation and instability
However, beyond these positive effects of immigrants at the firm level there are also positive external effects at the community level. These effects are also included in Table 1. Positive externalities include the role of cultural diversity as an amenity: an increased demand for ethnic goods and services in the community which local firms can aim to satisfy. Additionally, local population growth through immigration contributes to agglomeration advantages, greater aggregate demand and additional gross fixed capital formation, with new technology embodied in new capital. Diversity may also improve community cohesion when bridging-type social capital formation leads to cross-cultural cooperation. Such positive externalities may contribute to an innovative ‘milieu’.
On the other hand, the literature – such as reviewed by Alesina and La Ferrara (2005) – also points to a range of potentially detrimental effects of ethnic or cultural diversity, both at the firm level and as an externality in the community. A negative impact of migrant diversity is the possibility of fractionalization: cultural and language differences can lead to communication problems, less trust, greater potential for conflict among staff, and discrimination of minorities. Such conditions may hamper innovation. Moreover, the greater labor intensity of production, which is a rational response to lower wages paid to immigrant workers, may lead to lower capital investment in the short run (the substitution effect), although this can possibly be offset by firm expansion in the long-run (the output effect). Additionally, the spatial sorting of native born and immigrant workers at the community level can lead to greater spatial segregation. This may imply less cross-cultural relations and trade, and lower spatial mobility and knowledge transfers in the long run. Additionally, “within group” bonding-type social capital formation can lead to “between group” conflicts and polarization. Finally, diverse communities may exhibit political fragmentation and instability that discourage innovation.
To date, no empirical research has yet been able to separate these different channels of influence of cultural diversity on innovation. The results that are reported later in the paper must therefore be interpreted as providing evidence of a net effect, or balance of effects. It is clearly a challenge for future research to identify the importance of each of the channels described above and summarized in Table 1.
3. Empirical strategy
In this section, we briefly explain the approach used by Statistics Netherlands to sample firms in the 2000-2002 survey of innovation, called Community Innovation Survey (CIS) 3.5, which provides the anchor of our empirical strategy. We also provide details of how the sample was modified. Finally, we describe the econometric modeling methodology used.
To create the sample used for CIS 3.5, Statistics Netherlands selected firms from the General Business Register. Only firms with SBI (business activity code) 1 through 74, 90, 92 and 93 were included. The excluded codes refer mainly to public-sector and NGO-type of activities. A further selection was made based on firm size. Firms employing less than 10 persons were not included in the sample. Firms employing more than 50 persons were all included in the sample. For firms employing 10 to 50 persons, only a fraction was randomly selected into the sample. The size of this fraction depends on the SBI code and firm size. After the survey, a weighting factor is calculated per stratum. A stratum is defined on the basis of two indicators: the 2-digit SBI and firm size1.
Variable (n = 4582)
Firm is an innovator in 2000-2002
Firm innovated by means of new products in 2000-2002
Firm innovated by means of new processes in 2000-2002
Firm size (number of employees)
Firm is part of a group
Headquarters is abroad
Openness to change
Obstacles: Lack of personnel
Obstacles: Lack of technology
Prepared knowledge strategy
Share of foreign born
Share of 2nd generation immigrants
Unique number of countries of birth among firm employees
Fraction of employees aged 18-24*
Fraction of employees aged 25-34
Fraction of employees aged 35-44
Fraction of employees aged 45-54
Fraction of employees aged 55-64
Fraction of low-skilled employees*
Fraction of middle-skilled employees
Fraction of high-skilled employees
Fraction of employees in scientific occupations
where the dependent variable is one of the three innovation types mentioned above, and i stands for a firm, i = 1,2,…..N. Firm variables include firm size, the stock of human capital and a set of other control variables. Firms are more likely to be innovative if they are more export-oriented and internationally connected. We control for this by adding the location of a firm’s headquarters into the econometric modeling. Moreover, 22 macro-sector fixed effects account for sector-specific shocks and unobserved heterogeneity (identical with 2-digit NACE codes, see the Appendix).
We also utilize reported obstacles to the innovation process to account for the availability of innovation inputs3. Hence, we take account of whether a firm reports a lack of personnel or technology as a constraint to innovation. Long-term planning of a knowledge acquisition strategy is an important factor for the success of innovation activity. We therefore include knowledge acquisition strategy planning as another control factor in our estimations4. Finally, we use the extent to which firms declare to be open to change as an additional attitudinal variable. Jensen et al. (2007) argue that the organizational capabilities of firms impact on innovation, possibly as much as science and technology investments do.
These firm variables used in our estimations are common indicators of innovativeness at the firm level. We now take the literature one step further by accounting for the composition of employment. The employee features considered in the analysis include the ethnic, demographic and occupational characteristics of the workers. The age composition of a firm’s workforce, measured by the shares of specific age groups in total employment, is used to test whether more youthful firms are more innovative (see, e.g., Poot, 2008). Similarly, the shares of various skill categories in total employment are used to test the impact of skills on innovation. We use the ratio of the number of foreigners to the total number of employees per firm as an indicator of the firm’s overall ethnic structure. Additionally, we complement this ‘share of foreigners’ with measures of diversity in which the country of birth composition is explicitly taken into account.
in which s ji is the share of the group j (j = 1, …, N) in foreign employment of firm i. The diversity of a firm increases with an increasing value of the index. The index value can range between 0 (all migrants originate from the same country) and 1-1/N (there are an equal number of migrants from each of all N country groups).
in which U ji is a dummy variable that is equal to one when country j is represented in firm i and zero otherwise. Hence the value of Unique increases with the increasing number of countries represented in each firm. Its value ranges between 1 and N 6 .
Description of foreignness
Since foreign employees are central to our analysis, a clear definition of foreigners is essential. Our dataset allows us to observe the birthplace and country of citizenship of an employee, as well as the birthplaces of both parents of the same employee. During the life course, an employee may move from one country to another and obtain a second citizenship, or change citizenship. Moreover, countries may categorize non-natives in different ways. For example, The Kingdom Act on Dutch Nationality identifies a Dutch person according to the parents’ birthplace and/or the individual’s birthplace. Thus, a person in The Netherlands is called ‘allochtoon’ if that person was born abroad or at least one parent was born abroad. In our analysis a foreign employee is simply any employee who was not born in the Netherlands7. However, this definition excludes the foreign born children of Dutch-born parents (who would typically be Dutch expats or return migrants).
One may argue that an employee who entered the Netherlands at a very early age is likely to acquire skills of the host country like a native. Although we also observe acquired Dutch citizenship, we cannot unfortunately observe the year of entry to the host country. However, to consider immigrant integration, we identify second generation foreigners based on the birth places of the employees and their parents. Thus, if a person was born in the Netherlands and neither or at most one of the parents was born in the Netherlands, then we identify him or her as a second generation immigrant. We argue that if at least one of the parents was born abroad, the child would still identify, to an extent, with the foreign-born parent’s cultural background.
Addressing endogeneity issues
To assess the impact the foreign employment on a firm’s innovativeness one has to take into account that the share of foreigners among a firm’s employees may be a function of the firm’s innovativeness. The bias introduced by ignoring this reverse causality is not a priori clear. On the one hand, skilled migrants may be non-randomly sorted into more innovative firms that perhaps pay better or offer better career opportunities or actively promote diversity. On the other hand, less innovative firms could use relatively more traditional and labor intensive production methods and recruit relatively many less skilled workers, among whom relatively many less skilled migrant workers may be found. The impact of endogeneity of migrant employment is therefore an empirical matter which we address by applying instrumental variables (IV) estimation, in which we employ two instruments to account for the potential endogeneity of the share of foreigners in a firm’s employment. The share of foreigners measures employment diversity between migrants and natives. We do not instrument diversity among immigrants, because if there is any selection at all, immigrants would be generally keen to work with migrants from their own cultural background, not from other backgrounds. It is not likely that innovativeness of a firm is, at least in general, a good predictor of a potential employee finding his or her cultural background represented in the firm.
The literature suggests that the past stock of immigrants can help to identify the impact of the current stock. It is widely accepted that new immigrants follow ‘well-trodden paths’ in terms of their location preferences even when local economic conditions that attracted earlier immigrants would not have been the same as those today. Consequently, we use the natural logarithm of the ‘allochtoon’ population by municipalities in 1996 as the first instrument for the 2002 share of foreigners in any firm. The year 1996 is the earliest date for which the ‘allochtoon’ population of the Netherlands can be disaggregated to the municipal level.
Secondly, while thinking about an instrument for foreign labor sorting, it is crucial to make a distinction between production and consumption externalities related to the presence of immigrant communities. Immigrant labor is an input into production, which can have productivity effects. Ethnic goods and services are a result of consumption demand. Foreign labor tends to be concentrated in certain cities and precincts, including in the Netherlands, where it creates demand for ‘home goods and tastes’ (Clark, 2002; Shapiro, 2003; Florida, 2003). The migrants can import these products themselves or start producing them in the host country, for example, by opening restaurants and shops specialized in ethnic cuisines. Immigrants do go for lunch or dinner in “restaurants of their own nationality” and consider proximity of such restaurants a benefit of their residential location. A typical example that would apply to many cities is the presence of an ethnic precinct such as Chinatown. Ethnic precincts attract both the migrant group for consumption activities and employment, but also the native population for consumption activities. The native population will increasingly demand the same goods due to the demonstration effect and the increased culinary variety (e.g. Gabaccia, 1998; Bakens et al. 2013). However, there is no reason to expect that the innovativeness of firms determines the location of restaurants offering foreign cuisine, when we exclude in IV estimation the hospitality sector itself. We therefore use the number of foreign restaurants per 10000 population in municipalities in the year 1996 as another instrument to account for the sorting of foreign labor. The dataset comes from the HORECA database in the Netherlands8. Our identification strategy could break down if innovation is predominantly undertaken by young firms that locate in ethnically diverse areas with many foreign restaurants, for example in order to recruit foreign employees. However, the percentage of innovating firms (38 percent in our data) is much larger than the percentage of firms less than three years old (about 15 percent)9. based on OECD averages). Moreover, because the Netherlands has strict and complex residential and commercial zoning regulations, the location decisions of the vast majority of firms in our sample, mostly long established, can be credibly assumed to have been unrelated to the presence of foreign restaurants.
4. Description of the constructed dataset
Our research analyses Dutch firms which have responded to the 2000-2002 CIS in the Netherlands. Several micro datasets were combined to create the master dataset. All of the micro datasets were obtained from Statistics Netherlands under a confidentiality agreement10. The micro datasets that are used to create our master dataset are as follows: (i) the Social Statistics Database (Sociaal Statistisch Bestand) that is in turn composed of tax data (REOS) and Dutch municipality registrations (GBA); (ii) the Community Innovation Survey (CIS); (iii) the Dutch Labor Force Survey (EBB) (for details see the Appendix). By using this available information, we created a master dataset in which we can identify firm-level information about the employees, their ethnic origin and location at the municipality level, while we also have extensive knowledge about the characteristics of the firms.
The master dataset was created in three steps. Firstly, CIS was merged with REOS to identify the number of employees per firm by using firm ID as a key variable. Secondly, we merged this new data set with the municipality registrations, from which we obtain the birthplaces, and various other characteristics of employees. Finally, because none of these datasets include information on the occupation and professional background of employees, the CIS_REOS dataset was merged with the labor force survey EBB to retrieve skills information. However, this step reduces the data set from a census of firms to a survey of employees in Dutch firms. As a result of this final merge, 45 percent of the firms that are in the CIS 3.5 remain included in the master dataset. This was the only way to obtain the skills of the employees. However, given that the Labor Force Survey is a representative random sample of the Dutch workers, the final dataset remains broadly representative of Dutch firms (with 10 or more employees). To avoid employee characteristics being directly affected by the measured innovation, the variables on employee characteristics were lagged by two years (i.e. they refer to the year 2000), while innovation information has been reported at the end of 2002.
Employees who are under 18 years old, and those who earn less than €10 a month are excluded from the database. The REOS database reveals information about people who were active in the labor market on 31 December 2000. They are assigned to the firm they worked for on that date, irrespective of when they started to work in that firm, or if they changed jobs subsequently. If the worker was observed in the dataset multiple times (multiple jobs), the job with the longest job spell was selected. Personal characteristics (age, marital status, citizenship, etc.) are those corresponding with the most recent residential location (in 2000), and each individual is counted only once. If an employee changed address, the most recent location defined the household address. The same methodology was applied to the observations from the EBB survey.
Description of firms
The most common sector is Wholesale Trade and Repair (15 percent of firms), followed by Other Business Services (14 percent) and Construction (11 percent). Many of these firms are located in the large agglomerations; primarily in the western ‘Randstad’ region11, followed by the s’-Hertogenbosch-Maastricht corridor in the South-east, and around Groningen in the north. 38 percent of the 4582 firms report innovation with respect to at least one of the innovation categories and many times in multiple categories (see Table 1). Overall, 28 percent of firms report new product or services innovations and 21 percent report new process innovations.
Multinational firms are more likely to employ foreign workers, and larger firms are more innovative than small firms. In our dataset 83 percent of the firms’ headquarters are in the Netherlands, while the rest are predominantly in neighboring countries such as the UK, Germany, and France. As expected, firms in some sectors are more likely to operate internationally than in others. The largest number of firms with headquarters abroad can be found in the Mining and Quarrying, Chemicals, and Machinery and Equipment sectors, in that order.
Description of foreigners
5. Results of multivariate regression analysis
We begin with showing the effects of various employee diversity measures on the different innovation categories by OLS. Then we account for the sorting of foreigners into particular firms by means of instrumental variables estimation13. We also address specific theoretical concerns raised in sections 1 and 2. Firstly, we consider the integration hypothesis through taking second generation immigrant employees into account. Secondly, we consider the immigrant skill composition and take a sub-sample that excludes the sectors that employ the highest shares of unskilled immigrants (retail trade, hotels and restaurants, and low-skilled business services, such as cleaning, security and secretarial services).
OLS Regression analysis with the full sample
Share of foreign born
Empl. in scientific occupations
Employees aged 25-34
Employees aged 35-44
Employees aged 45-54
Employees aged 55-64
We now focus on the employee variables in Table 3. We find that the presence of highly-skilled employees appears to be an important determinant of innovation as a whole. In particular, firms with a highly-skilled or scientifically trained workforce have significantly more product innovations at the 1 percent and 5 percent level respectively14. However, for process innovations we find a statistically significant effect of only the scientifically trained workforce (at the 5 percent level).
The results regarding age composition of the employees of the firm show that a more youthful workforce generates more innovation. Among the four age categories included in the regressions, the age groups 25-34 and 35-44 turn out to have a very considerable effect on all innovation categories. Interestingly, a statistically significant positive effect is also found for employees aged 55-64 (relative to the default age category 18-24) in the case of innovativeness as a whole. However, the coefficient is smaller than the corresponding coefficients for employees aged 25-34 or 35-44.
Table 3 shows that a greater share of foreigners in employment is associated with lower innovation as a whole and also with less product innovations. A similar result was also found by Ozgen et al. (2012) in an analysis of patent applications at the regional (NUTS 2) level in Europe. Brunow et al. (2012) found a similar effect with respect to firm productivity, using micro-level firm data from Germany. As noted in Section 2, the negative effect of a relatively larger share of foreigners in overall employment is plausible when one considers that the reservation wages of the migrant workers are likely to be lower than those of the native born population. Consequently, by recruiting more foreign workers, firms lower the cost of labor relative to capital. This induces more labor-intensive production, which is less likely to be encouraging innovation than capital-intensive production technologies.
However, the benefit from migration for innovativeness of the firm comes instead from the diversity of the foreign workforce. We find that the diversity index has a positive effect on product innovations and on innovations in general. Both are statistically significant at the 5 percent level, while the effect is statistically insignificant for process innovations. The result suggests that the cultural composition of foreign employment certainly matters for a firm’s innovativeness.
The second measure of diversity we utilize is the natural logarithm of a measure called ‘unique’, which stands for the total number of unique birthplaces in each firm, as explained in Section 3. This is quite a different way of measuring diversity than accounting for the fractionalization of foreign employment in a firm although the two measures are correlated (the correlation coefficient is 0.60). We find that, for product innovations, an increasing number of birthplaces increases the probability to innovate, though the effect is significant only at the 10 percent level.
IV Estimations with the full sample
Share of foreign born
Share of foreign born
Share of foreign born
(1), (2), (3)
(4), (5), (6)
(7), (8), (9)
Number of foreign restaurants
per 10,000 population in 1996
Log of ‘allochtoon’ in 1996
Share of foreign born
Share of 2nd generation
Hansen J statistic
Together, the two instruments pass the formal Hansen J test of overidentification, and turn out to be strong instruments with F values greater than 10 in estimations with the full sample reported in Table 4. The F value for the instrument number of foreign restaurants is slightly less than 10 when the IV estimations are repeated for the sample of firms where sectors employing low-skilled foreigners are excluded (see below).
With respect to the negative coefficient for the share of foreigners in firm employment, one can argue that not all of the foreign born should be considered the same. Those who spent more time in the host country are likely to have acquired some native skills and language. Therefore, they should not trigger the same innovation decreasing effects as newcomers. Although we cannot identify the date of entry of a foreigner into the Netherlands, we observe the parents’ and the employee’s birthplaces. Therefore, by including second generation immigrant workers into the IV regressions reported in Table 4, we can test whether there is a difference between first generation immigrant workers and those who were born in the Netherlands in their contribution to firm innovations. The results in Table 4 show that the impact of the share of first generation foreigners is negative and significant for innovations as a whole and for product innovations. The coefficient for product innovations is smaller than that for all innovations. However, we find that relative employment of second generation immigrants does not significantly impact on innovation at all, or with a coefficient that is much less than for first generation immigrants (see columns (7) and (9)). Unfortunately, the data do not allow us to scrutinize the specific mechanisms that may cause these results.
When we add in Table 4 the diversity index to the share of foreigners, we find that the diversity index is positive and significant (at the 1 percent level) for innovations as a whole and for product innovations. The effect size is relatively higher than those estimated in other specifications. The other covariates included in all of the IV estimations are quite robust with respect to previous findings.
IV Estimations when sectors employing predominantly low-skilled foreign workers are excluded
Share of foreign born
Share of foreign born
Share of foreign born
(1), (2), (3)
(4), (5), (6)
(7), (8), (9)
Number of foreign restaurants per 10,000 population in 1996
Log of ‘allochtoon’ in 1996
Share of foreign born
Share of 2nd generation
Hansen J statistic
Estimation is again presented by means of IV. With this subsample, the share of foreigners remains statistically significant for innovations as a whole and for product innovations. Once sectors employing predominantly low-skilled immigrants are left out of the sample, the coefficients of the diversity variable are larger. Moreover, the ‘unique’ variable is also positive and statistically significant, both for innovations as a whole and for product innovations in Table 4 and even for process innovations in Table 5. Therefore, we conclude that a skilled workforce and a rich birthplace composition of employment are important factors in increasing the probability to innovate.
We also explored the relationship between R&D orientation and cultural diversity. The Dutch CIS survey categorizes firms into two types: firms with no R&D activity, and other firms. To measure the impact of worker diversity in firms with some kind of R&D activity, we drop all firms with no R&D orientation. For the sake of brevity we do not report our results in a separate table. The results are qualitatively the same as those in Tables 3, 4 and 5.
It should be recognized that the effects identified in this paper are quantitatively small. An increase in the diversity index from 0.1 (a third of a standard deviation) below the mean (0.45) to 0.1 above the mean increases the probability to innovate products by 2 percentage points in the full sample and by 3 percentage points in the restricted sample. An increase in the unique number of birthplaces similarly increases innovation by 0.1 percentage points in the full sample and 0.15 percentage points in the restricted sample16. An increase in the share of foreigners in a firm’s employment by 1 percentage point above its mean of 0.1 (which is quite a large increase) decreases the predicted probability to innovate by 2 percentage points at the most. Overall, we observe that when foreigners account for a large share of employment, a firm is less likely to report innovative activities, but the effect is quantitatively small. However, diversity among foreign workers is a booster of innovation. Finally, we analyzed interaction effects between the diversity of the workforce and the share of foreigners in employment. The interaction effects are statistically insignificant, while the positive and significant impact of workforce diversity persists for product innovations. Moreover, the negative effect of the share of foreigners persists as well17.
In this paper we focused on estimating the impact of immigrant employees on the innovativeness of Dutch firms. The core data on innovation were obtained from the Community Innovation Survey (CIS) of 2002. However, this survey provides little information on the size and composition of the workforce of the firms. Such information was obtained by linking the CIS with administrative data; namely, the Social Statistics Database, which – in turn – is made up of tax data, and population registration data at the municipality level. While this linked dataset provided measures of the presence and diversity of migrants, it yielded no direct information on the skill level of the migrants. The latter was obtained by linking the data to the Dutch Labor Force Survey (EBB). Because the EBB is a representative sample survey, but not a census, many observations are lost in the linking process. The merging of the firms surveyed in the CIS with the employees surveyed in the Dutch Labor Force Survey was a major research challenge. The final sample consisted of 4582 firms, employing around 1.3 million workers (and around 15,000 workers with known occupations) of whom about 13 percent are foreign born.
The results of the regression analysis concerning the internal resources of firms for innovation are in line with the literature. Additionally, our results emphasize that ‘soft’ factors like firms’ openness to institutional change or development of a knowledge strategy help to reap fruits of R&D in terms of implementing new product and process innovations. The regressions showed that such activity requires the relative abundance of medium and high-skilled employees. In terms of age composition, a relative abundance of employees aged 35-44 boosts innovativeness.
The main purpose of the paper was to investigate whether the explicit presence and diversity of migrants boosts innovation. Generally speaking, a larger share of foreigners among a firm’s employees lowers the innovativeness of firms. Moreover, once endogeneity is controlled for by means of valid instruments (the past density of foreign restaurants per municipality, and the past presence of migrant communities in the municipality), the negative and statistically significant impact of the share of foreign workers on the firm’s innovativeness is confirmed and even larger. This finding is, of course, consistent with neoclassical economic theory that predicts that when there is an abundance of migrant workers with lower reservation wages; firms will develop expansion strategies that involve the use of labor-intensive production technologies. This is likely to imply that fewer resources, including those linked to new capital equipment, will be devoted to innovation. In any case, this negative impact is shown to be less for second generation immigrants. On the other hand, there is evidence that greater diversity among foreign workers stimulates innovation in general. This is the case whether diversity is either measured by the unique number of foreign countries of birth represented among the firm’s staff or by a fractionalization index. The positive impact of diversity on innovation is more robust across specifications when focusing on product innovations than on process innovations. However, the positive impact of immigrant diversity on innovation is quantitatively modest; a one standard deviation (0.3) increase in the diversity index raises the probability that the firm is an innovator by 3 to 4.5 percentage points (around the mean of 38 percent).
Despite the considerable effort that was required in constructing the data set for the analysis, the data did not permit us to ascertain precisely what kind of attributes of a diverse set of migrants may boost innovation, nor which of the various transmission channels outlined in Figure 1 dominates. Future research would benefit from setting up a panel data set that may account for unobserved firm heterogeneity and from obtaining additional migrant characteristics from administrative registers that may provide information much larger micro data sets on migrants. Moreover, as a first step to disentangling the channels of influence outlined in the paper, research ought to try to separate out within firm effects from external spillovers in the local labor market and community.
Nevertheless, the results of our current study already provide some tentative lessons for policy. Firstly, immigrants from a range of countries are neither a homogenous group of workers nor perfect substitutes for locally born workers. Their presence impacts on the innovativeness of host firms dependent on the kind of human capital they provide. Secondly, as long as necessary conditions are present (such as scale, infrastructure, networks and institutional openness of the firms), diversity of the workforce seems to benefit particularly the sectors in which diverse skilled migrants cluster. Finally, it is crucial for firms to design policies to attract high-skilled immigrants from a range of cultural backgrounds in order to compete innovatively in the global marketplace.
1 To correct for a possible sample selection issue, we reweight all of our observations by using the total sum of weights for each stratum in the CIS 3.5 based on the two criteria given in the text. Therefore, it is possible to compare the distribution of firms in our sample with that of the population. This exercise reveals that our estimations are based on a sample in which firms with more than 50 employees are somewhat overrepresented. For that reason, the results are more representative of the impact of foreigners on large firms, rather than capturing their impact on small firms. Additionally, non-responses are treated as an absence of innovation. This imputation makes little difference.
2 The exact definitions of the dependent variables are as follows: a product innovation is the market introduction of a new good or service or a significantly improved good or service with respect to its capabilities, such as improved software, user friendliness, components or sub-systems. A process innovation is the implementation of a new or significantly improved production process, distribution method, or support activity for a firm’s goods or services. If a firm aborted an innovation during the study period, the firm is still counted as having been innovative.
3 Some input variables, such as total R&D expenditures, predict innovation activity perfectly. In other words, all firms in our sample that reported R&D expenditures between 2000-2002 were innovative.
4 The detailed description of the variables used in the estimations is given in Table 6 in Appendix.
5 The diversity index is sensitive to the presence of dominant migrant groups among all foreign born. Consequently, we measure the diversity index by dividing the world into 12 supra-national regions: Anglo, Confucian Asia, Eastern Europe, Germanic Europe, Latin America, Latin Europe, Middle East, Nordic Europe, Southern Asia, South Eastern Europe, Sub-Sahara Africa, and Rest of the World. This categorization, referred to as GLOBE Clusters in the literature (Gupta et al. 2002) is based on the cultural distances between countries. N is 12 in equation (3), but about 200 in equation (4).
6 The number of countries of birth represented in a firm varies between 1 (Netherlands only) and 197. The average is ten countries (see Table 2).
7 Although we also observe acquired Dutch citizenship, time spent in the host country is not known.
8 HORECA is an acronym for Hotels, Restaurants and Cafes. This dataset covers the registered bars, restaurants, hotels and all the other businesses in this sector. The data coverage is fairly good and it is currently available from 1995 until 2007. The number of foreign restaurants is counted at the municipal level by the nationality of the cuisine, at the time of registration. For each restaurant, the main location is registered; no ‘side’ kitchens are registered. This leads to a unique listing, and excludes double counting.
9 The age of a firm is not recorded in our dataset but the estimate is based on OECD data.
10 Access to the data is restricted to certain locations and data terminals.
11 The Randstad region is a large conurbation which is composed of four major Dutch cities (Amsterdam, Rotterdam, The Hague, and Utrecht) and their hinterlands. The region generates 46 percent of national GDP and houses 41 percent of the population in the Netherlands.
12 Among those are the fall of the Iron Curtain, the unification of East and West Germany, as well as prolonged wars in the Balkan area.
13 All the estimations are carried out with Stata 11.
14 The skill categorization follows the International Standard Classification of Occupations. Since the main focus of this paper is the contribution of the foreign workers to innovation we also replaced the employee characteristics regarding skill and age with the corresponding variables for foreign employees. While the age group variables for foreigners are robust to the reported findings in Table 3, we do not find an impact of any of the skill category-based foreigners’ shares in firm employment. This is due to the small variation in the number of foreign people in these skill categories. This stems possibly from the fact that the skill data come from the Dutch Labor Force Survey where we observe much smaller number of foreigners in comparison with municipal registrations.
15 Cleaning, security, call centres, secretaries, fotography developers, etc.
16 Ozgen et al. (2012) showed, by using a panel data of 170 NUTS 2 level in twelve European countries, that a 0.2 increase around the mean of the diversity index (0.49) increases the regional patent applications by about 4 percent.
The description of the firm variables used in the estimations
Firm related variables
Description of the variables
Firm is an innovator in 2000-2002
A dummy variable equal to ‘1’ if the firm reported any innovation in 2000-2002
Firm innovated new products in 2000-2002
A dummy variable equal to ‘1’ if the firm innovated new products/services or significantly improved existing products/services in 2000-2002
Firm innovated new processes in 2000-2002
A dummy variable equal to ‘1’ if the firm innovated new processes or significantly improved the existing processes in 2000-2002
Number of sampled employees per firm
Number of employees in a firm
Firm is part of a group
A dummy variable equal to ‘1’ if the firm is part of a company group
Headquarters is abroad
A dummy variable equal to ‘1’ if the headquarters of a firm are located outside of the Netherlands
Openness to change
A dummy variable equal to ‘1’ if the firm has applied organizational changes with respect to its operations in 2000-2002
Obstacles: Lack of personnel
A Likert scale from 1-7; 0 = no problem, 1 = minor, and 7 = severe: Lack of personnel has been reported as an obstacle to innovations of a firm in 2000-2002
Obstacles: Lack of technology
A Likert scale from 1-7; 0 = no problem, 1 = minor, and 7 = severe: Lack of technology has been reported as an obstacle to innovations of a firm in 2000-2002
Prepared knowledge strategy
A Likert scale from 0-2; 0 = No strategy, and 2 = Firm has a strategy: Firm has prepared a knowledge management strategy in 2000-2002
Social Statistics Survey
The unit records of the Social Statistics Survey database are very detailed and informative about 10 million jobs per year (a job is a matched combination of employer/business entity data with employee data and recorded start/end date, if applicable). The job data include the location of residence and work of an employee, information on the labor market and income level. Municipality registrations (GBA) are a natural extension of the Social Statistics Survey provided by Statistics Netherlands, and allow us to calculate the exact number of employees per firm at the municipal level, when combined with the tax data (REOS).
Community Innovation Survey (CIS)
The CIS is conducted every two years (in even years). In this study we use the CIS 3.5 version (2000-2002), where the final reporting year is 2002. This version of the survey includes 10,533 firms. The response rate of firms in the CIS 2000-2002 survey is 62 percent. Each firm has a firm identification number ‘BEID’. The continuity of surveys provides a regular snapshot of various aspects of innovation. It is not only informative about technological innovations on new products/services and goods, but also includes a certain amount of information on non-technological/business/organizational changes and newness. The survey allows us to classify firms by 5-digit ‘SBI93’ codes that stand for Standard Industrial Classification of Statistics Netherlands, and these codes are directly analogous with the 2-digit NACE codes. These are: Agriculture, Forestry and Fishery; Mining and Quarrying; Food, Beverage and Tobacco; Textile, Clothes and Leather; Wood, Paper and Pulp; Chemicals; Metals; Machinery and Equipment; Manufacturing n.e.c.; Electricity, Gas and Water; Construction; Wholesale Trade and Repair; Retail Trade; Hotels and Restaurants; Transport and Communication; Financial Intermediation; Real Estate and Renting of Machinery; Computer and Related; Research and Development; Other Business Services; Environmental Services; and Other Services.
Dutch Labor Force Survey
The Dutch labor force survey (EBB) is a quarterly survey, conducted since 1996, with a rotating panel drawn from the Dutch labor force. A single respondent in the panel is expected to be surveyed 5 times in total, over the course of 15 months. The EBB contains a variable called ‘RIN’ that stands for the personal identification number which allows researchers to combine it with various datasets such as the tax data and municipality registrations by using this id. The survey includes a fairly extensive number of questions in 28 categories regarding households as well as Dutch and foreign employees. Some of the modules in the EBB Overview are as follows: Household, Ethnicity, Birth Country, Service (type of contract), Working Hours, Commuting, Company, Job Situation, Job Searching, Trade Union, Education, and Pension.
This research is part of the Migrant Diversity and Regional Disparity in Europe (MIDI-REDIE) project, funded by the NORFACE research program Migration in Europe – Social, Economic, Cultural and Policy Dynamics. We are grateful to Henri de Groot, Dave Maré, other members of the MIDI-REDIE team, participants at the 50th European Congress of the Regional Science Association International, Jönköping, Sweden, 19-23 August 2010, and at the international conference Migration: Economic Change, Social Challenge, University College London, 6-9 April 2011, for comments on earlier versions of the paper.
Responsible editor: Amelie F Constant
- Acs ZJ, de Groot HLF, Nijkamp P: The Emergence of the Knowledge Economy: A Regional Perspective. Berlin: Springer Verlag; 2002.View ArticleGoogle Scholar
- Agrawal A, Kapur D, McHale J: How do spatial and social proximity influence knowledge flows? Evidence from patent data. J Urban Econ 2008,64(2):86–111.View ArticleGoogle Scholar
- Alesina A, La Ferrara E: Ethnic diversity and economic performance. J Econ Lit 2005, 43: 762–800. 10.1257/002205105774431243View ArticleGoogle Scholar
- Alesina A, Devleeschauwer A, Easterly W, Kurlat S, Wacziarg R: Fractionalization. J Econ Growth 2003, 8: 155–194. 10.1023/A:1024471506938View ArticleGoogle Scholar
- Almeida P, Kogut B: Localization of knowledge and the mobility of engineers in regional networks. Manag Sci 1999, 45: 905–916. 10.1287/mnsc.45.7.905View ArticleGoogle Scholar
- Alvarez J, Forrest D, Sanz I, Tena JD: Impact of importing foreign talent on performance levels of local co-workers. Labour Econ 2011, 18: 287–296. 10.1016/j.labeco.2010.11.003View ArticleGoogle Scholar
- Audretsch DB, Feldman MP: Knowledge spillovers and the geography of innovation. Handb Reg Urban Econ 2004, 4: 2713–2739.View ArticleGoogle Scholar
- Bakens J, Mulder P, Nijkamp P: Economic impacts of cultural diversity in The Netherlands: productivity, utility, and sorting. J Reg Sci 2013,53(1):8–36. 10.1111/jors.12012View ArticleGoogle Scholar
- Brunow S, Südekum J, Trax M: Cultural diversity and plant productivity. Norface Migration Discussion Paper No. 2012–29; 2012.Google Scholar
- Caragliu A, Nijkamp P: The impact of regional absorptive capacity on spatial knowledge spillovers: The Cohen and Levinthal model revisited. Appl Econ 2012,44(1):1363–1374.View ArticleGoogle Scholar
- Clark TN, Lloyd R, Wong KK, Jain P: Amenities drive urban growth. J Urban Aff 2002, 24: 493–515. 10.1111/1467-9906.00134View ArticleGoogle Scholar
- Cohen WM, Levinthal DA: Absorptive capacity: a new perspective on innovation and learning. Adm Sci Q 1990, 35: 128–152. 10.2307/2393553View ArticleGoogle Scholar
- Döring T, Schnellenbach J: What do we know about geographical knowledge spillovers and regional growth?: A survey of the literature. Reg Stud 2006,403(3):375–395.View ArticleGoogle Scholar
- Faggian A, McCann P: Human capital flows and regional knowledge assets: A simultaneous equations approach. Oxf Econ Pap 2006, 52: 475–500.View ArticleGoogle Scholar
- Florida R: Cities and the creative class. City & Community 2003, 2: 1.View ArticleGoogle Scholar
- Gabaccia DR: We Are What We Eat: Ethnic Food and the Making of Americans. Harvard University Press; 1998.Google Scholar
- Grossman GM, Helpman E: Endogenous innovation in the theory of growth. J Econ Perspect 1994,8(1):23–44. 10.1257/jep.8.1.23View ArticleGoogle Scholar
- Gupta V, Hanges PJ, Dorfman P: Cultural clusters: methodology and findings. J World Bus 2002,37(1):11–15. 10.1016/S1090-9516(01)00070-0View ArticleGoogle Scholar
- Hunt J, Gauthier-Loiselle M: How much does immigration boost innovation? American Economic Journal: Macroeconomics 2010, 2: 31–56.Google Scholar
- Jensen MB, Johnson B, Lorenz E, Lundvall B: Forms of knowledge and modes of innovation. Res Policy 2007,36(5):680–693. 10.1016/j.respol.2007.01.006View ArticleGoogle Scholar
- Jones CI, Romer PM: The new Kaldor facts: ideas, institutions, population, and human capital. Am Econ J 2010,2(1):224–245.Google Scholar
- Kerr WR: Breakthrough inventions and migrating clusters of innovation. J Urban Econ 2010, 67: 46–60. 10.1016/j.jue.2009.09.006View ArticleGoogle Scholar
- Kerr WR, Lincoln WF: The supply side of innovation: H-1B visa reforms and U.S. ethnic invention. J Labor Econ 2010,28(3):473–508. 07 10.1086/651934View ArticleGoogle Scholar
- Lee N, Nathan M: Knowledge workers, cultural diversity and innovation: evidence from London. Int J Knowl-Based Dev 2010,1(1–2):53–78.View ArticleGoogle Scholar
- Lobo J, Strumsky D: Metropolitan patenting, inventor agglomeration and social networks: a tale of two effects. J Urban Econ 2008, 63: 871–884. 10.1016/j.jue.2007.07.005View ArticleGoogle Scholar
- Mairesse J, Mohnen P: Using innovation surveys for econometric analysis. In Handbook of the Economics of Innovation. Edited by: Hall BH, Rosenberg N. Amsterdam: Elsevier; 2010:1130–1155.Google Scholar
- Mazzolari F, Neumark D: Beyond wages: The effects of immigration on the scale and composition of output. BER Working paper 14900, Cambridge Mass; 2009.Google Scholar
- Niebuhr A: Migration and innovation: does cultural diversity matter for regional R&D activity? Pap Reg Sci 2010, 89: 563–585. 10.1111/j.1435-5957.2009.00271.xView ArticleGoogle Scholar
- OECD: International mobility of the highly-skilled. Policy Brief (July 2002). 2002. http://www.oecd.org/science/inno/1950028.pdf Google Scholar
- Ozgen C, Nijkamp P, Poot J: Immigration and innovation in European regions. In Migration Impact Assessment: New Horizons. Edited by: Nijkamp P, Poot J, Sahin M. Cheltenham, UK: Edward Elgar Publishing, Inc; 2012.Google Scholar
- Page S: The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton University Press; 2007.Google Scholar
- Parotta P, Pozzoli D, Pytlikova M: The nexus between labour diversity and firm’s innovation. Norface Migration Discussion Paper No. 2011–5 2011. http://www.norface-migration.org Google Scholar
- Poot J: Demographic change and regional competitiveness: the effects of immigration and ageing. Int J Foresight Innovation Policy 2008,4(1–2):129–145.View ArticleGoogle Scholar
- Romer PM: Endogenous technological change. J Polit Econ 1990,98(2):S71-S102.View ArticleGoogle Scholar
- Saxenian AL: The New Argonauts: Regional Advantage in a Global Economy. Cambridge, MA: Harvard University Press; 2006.Google Scholar
- Shapiro JM: Smart cities: explaining the relationship between city growth and human capital. NBER Working paper (June 19); 2003.Google Scholar
- Simonen J, McCann P: Firm innovation − The influence of R&D cooperation and the geography of human capital inputs. J Urban Econ 2008,64(1):146–154. 10.1016/j.jue.2007.10.002View ArticleGoogle Scholar
- Stuen ET, Mobarak AM, Maskus KE: Skilled immigration and innovation: evidence from enrolment fluctuations in US doctoral programmes. Econ J 2012, 122: 1143–1176. 10.1111/j.1468-0297.2012.02543.xView ArticleGoogle Scholar
- Südekum J, Wolf K, Blien U: Cultural diversity and local labour markets. IZA Discussion paper 4619; 2009.Google Scholar
- Wimmer A: The making and unmaking of ethnic boundaries: a multi-level process theory. Am J Sociol 2008,113(4):970–1022. 10.1086/522803View ArticleGoogle Scholar
- Zucker LG, Darby MR: Star Scientists, Innovation and Regional and National Immigration. In Entrepreneurship and Openness Theory and Evidence. Edited by: Audretsch DB, Litan RE, Strom R. Cheltenham, UK: Edward Elgar Publishing, Inc; 2009.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.