Loading...
HomeMy WebLinkAbout2019-08-19 Board of Health Packet ' 3 Meeting Posting with Agenda ;a 3 N CLE �, TvNtd CLERK REAMNIC MA. Board - Committee - commission - Council: Board of Health 2115 AUG 14 AM 10: 56 Date: 2019-08-19 Time: .6:30 PM Building: Reading Town Hall Location: Select Board Meeting Room Address: 16 Lowell Street Agenda: Purpose: General Business - Meeting Called By: Laura Vlasuk -Healt Agent on behalf of Kevin Sexton Notices .and agendas are to be posted 48 hours in advance of the meetings excluding Saturdays, Sundays and Legal Holidays. Please keep in mind the Town Clerk's hours of operation and make necessary arrangements to be sure your posting is made in an adequate amount of time. A listing of topics that the chair reasonably anticipates will be discussed at the meeting must be on the agenda. All Meeting Postings must be submitted in typed format: handwritten notices will not be accepted. Topics of Discussion: 630 pm -Open to public discussion (please sign in and state your name if speaking) 6:35 pm -BOH Reorganization -Health Agent Report -Discussion of monthly Health Agent Report to include a detailed spreadsheet that will be reviewed at every meeting of violations cited during inspections and link spreadsheet to the Health page on Town website -Review proposed pesticide regulations in tree lawn policy -Mass. Department of Public Health CBD advisory -Review minutes 6-18-19 - This Agenda has been prepared in advance and represents a listing of topics that the chair reasonably anticipates will be discussed at the meeting. However the agenda does not necessarily Include all matters which may be taken up at Nis meeting. Page I t f r'y Town of Reading +, All, a, Meeting Posting with Agenda t ` r 1 b_ ii vor i yaJ Ci ERI( p� SR i_Rut . 'Ak. Board - committee - commission - council: 2iS9 JUL 24 PPS 3; 16 Board of Health Date: 2019-08-19 Time: 6:30 PM Building: Reading Town Hall Location: Select Board Meeting Room Address: 16 Lowell Street Agenda: Purpose: General Business Meeting Called By: Laura Vlasuk -Healt Agent on behalf of Kevin S6.xton Notices and agendas are to be posted 48 hours in advance of the meetings excluding Saturdays, Sundays and Legal Holidays. Please keep in mind the: Town Clerk's hours of operation and make necessary arrangements to be sure your posting is made in an adequate amount of time. A listing of topics that the chair reasonably anticipates will be discussed at the meeting must be on the agenda. All Meeting Postings must be submitted in typed format; handwritten notices will not be accepted. Topics of Discussion: - 6:30 pm - Open to public discussion (please sign in and state your name if speaking) 6:35 pm A - Health Agent Report V - Discussion of monthly Health Agent Report to include a detailed spreadsheet that will be - reviewed at every meeting of violations cited during inspections and link spreadsheet to the Health page on Town website - Review proposed pesticide regulations in tree lawn policy -Mass. Department of Public Health CBD advisory -Review minutes 6-18-19 PA This Agenda has been prepared in advance and represents a listing of topics that the chair reasonably anticipates will be discussed at the meeting.However the agenda ones not rr,,sSarily include all matters which may be taken up at this meeting. page I 1 Health Agent update from June 1st to July 31, 2019 • Inspections • Food inspections/re-inspection 23 in June, 14 in July • complaints/animal inspections/dumpster 13 Complaints, 11 have been inspected and corrected, 2 are still out going 0 Animal inspection • Septic Abandonment 1 location • Community Health/Public Health Nursing Division • Flu shots administered 0 administered Upcoming flu clinics Update on flu clinic at Fall Street fair • Maven 4 cases in June, 4 cases in July • Tobacco update no violations Fine Asian Comra • Seehi Bar July 17,2019 To the Board of Health We are a local business sinceA997.After reviewing 6/18 meeting on YOUTube, my comments as following: This is all about public record transparency and easily accessible to BOH when needed, Through times,never be a single town resident have ever requested except BOH The great goal for BOH is, no question, but to protect public health and safety.But upfront, neither in irritating nor interfering to local business growth, or to create any unpleasant elements from attracting businesses and consumers to this great Town of Reading. This is unwisely to post any violation data online or into those uncertain social media,while Reading Planning Board works hard day by day to race, to compete, and to fight the surrounding towns in economic growth. Tremendous resources and efforts that both Town Manager, Robert LeLacheur and Assistant Town Manager,Jean Delios have already put out to achieve a brighter future for our town, no local businesses should be suffering from this unworthy idea, to which in preventing us from moving forward. Since the public may able to access all data in many ways anyhow, why bother to impact ourselves in negative and unworthy way.And to send our local businesses to the neighborhood towns. Herein, I would like to ask before the Board to withdraw this idea. I ect, Robert Lee Mandarin Reading 296 Sol..Street Reading,W 01867 781.9428200 6-6aogw mm.com J a Town of Reading e Meeting Minutes OA4•P* Board - committee - Commission - Council: Board of Health Date: 2019-06-18 Time: 6:30 PM Building: Reading Town Hall Location: Berger Room Address: 16 Lowell Street Session: Open Session Purpose: General Business Version: Draft Attendees: Members - Present: " nom"' Kevin M. Sexton, Eleanor Tate Shonkoff, Emmyr , Lara Romanowski Members - Not Present: Others Present: Jean Delios, Health Agent Lau fi suk Minutes Respectfully Submitted B Lara Roma o Topics of Discussion: Sexton opened meeting at 6:35 iA. •^ Chair Report vr. • Thanked Heidi er for her s ' ,e as ass ate board member; her term has ended. Heath Agent Report - • Summarized key 4 n ins c ion's(see attached) Discussio aft in on spreadsheet off: Spiea- , et is - at start; also seeking feedback from the business community. •. omanowski: Let' simp at is the purpose of this,and what are we trying to solve foo. • ton: Need to prate ublic h i the public can already access inspection reports at town hall or a specific establi ent. Doe n't believe posting the establishment names is necessary. • Via ero foodborne reeks or illnesses in the past two years. • Dove: pose of t readsheel is for the Board of Health to track inspection activity. It could also be u are schanism for public consumption which will have some of that information but not all. Io have establishment names in the spreadsheet.There's plenty of towns that post establishme aures. • Shonkoff. Doesn't feel the information is readily accessible to the public. Unclear on level of detail that will be available to the public in spreadsheet. Opening up the discussion to public commentary • Diane Manahan, Pamplemousse: How often do people in Reading look for reports? • Vlasuk: In two years,only Reading Board of Health members have asked for reports. • Ad, Columba Pizzeria: This was done for 1-2 weeks about 5 years ago; people didn't like it. • Chris, Cafe Nero: The 2013 food code allows for a sign in front of establishment describing that customers can request inspection report. There's different level 1,2, 3's.The businesses are more Page 1 1 cos or x£qe ° Town of Reading � = Meeting Minutes concemed with the customers than the Board of Health is because they don't want to lose a customer. We all want to protect customers. • Dove: Could we consider rolling out spreadsheet to only the Board of Health first? • Kristin, Food Services Director School Town of Reading: 1)If there's no opportunity to describe violation,then just opening up for criticism that is unwarranted. The violation isn't the problem, it is the ability to understand what is being done to prevent it in future 2)Any documentation that the Board of Health uses is a public record, so cautions on whatever decision is made, it may cause unintended consequences. • Michael Palmer, Fusilli's:Was very upset with the BOH. Ifs going to �1Reading very non- competitive, especially due to social media, lots of misinterpretation,,.' town,economically,you will dis-encourage people from going to establishments, businesses -eing publicly shamed. Nothing wrong with existing process.Would prefer to pull health licen n , acing this information for sharing on social media, • Carmen,Venation Moon: It seems like the Board is looki�gifor a solution to blem that doesn't exist.There is no problem.All information is availab d the public Its going t t city town a lot of money,and doesn't make any sense. • Elish, Bunratty Tavern:The most important this"two keep s n p their Comers ha a G happy althy. Putting this information online will cause mass hysteria in busine • Diane Manahan, Pamplemousse: Some health cod o alio a very minor, and people don't understand the details, Some peopl ave nothing bet than slander businesses. • Michael Palmer.Small businesses - handcuffed. • Lisa Eagan, Executive Director of C mercer wi I d statement in packet. (attached) • Bill Cushing, Christopher's Resta ran eral n for 2 s.This will create a problem, not solve one.There isn't a problem. `> .( • Sexton:This 3-perso messy du op `eti , Bear with us as we work through this,it is the first ti the op e of you,thal'Swhere we are tonight. It doesn't seem like we're ea to come o conclusion onight. • Shonkoff,No aI y to make a Sion tonigh nt to take feedback from tonight into account. • Dove:Agree, n dy to ma cision tonig, • Emily, Bagel Wor uld o e . ns,their size, location,and feedback on output I of i B portin specific a ing why it's a good idea. Reading is not New York, «5 1 e break» ry Review o lawn policy '7G • As air of the Sale, card to send us all their feedback to discuss at the next Board of Health meed �. Review minutes 5121119-Minutes acce amended. 423119-Minutes accept Sexton adjourned meeting at 8:05pm. Page 1 2 Health Agent update from May list t May 31, 2019 • Inspections • Food inspections/re-inspection 44 • complaintslanimal inspections/dumpster 2 Complaints all 2 have been inspected and corrected 0 Animal inspection • Septic Abandonment 2 locations • Community Health/Public Health Nursing Division • Flu shots administered 0 administered in May • Maven 0 cases • Tobacco update no violations • Health Agent Report Review of spreadsheet that Paul, a Reading resident provided Heidi —feels Paul's spreadsheet is great the way it is Eleanor -would like establishment names - specific violations added discussed at meeting and posted to website searchable by the public Emmy - would like the exact code violation listed - .track trends. This would make it easier to see how frequently a business is making the same mistake or if many businesses are making the same mistake - Likes using the code for the location but wants the code that references the name of the restaurant attached � f x ƒ g ƒ § 7 � . / It bo � ) � \ ° § / j ] 7 | • < � . % a f a ,j \ } u a / � \ fyt / . q \ ` � \�zi � \ • / / / / ` / � � . . � m J 0 r a N N d L a Ol _ -- - ---------- 2 Lu c 00 C Y d cd G 0 N m Y � ` v a J c bq 'v1 U1 (O C3 Z C Dear Board of Health: The Reading-North Reading Chamber of Commerce believes the proposed polity change which requires the BOH to post infractions found during Health Inspections online is overly unfriendly to our local businesses. The current BOH inspection process has been working successfully to both protect the public health and the business integrity of local restaurants. Currently, if there are serious issues found at a restaurant,there is a policy to notify the public. It is understandable in the restaurant business that there are occasional, minor issues found at inspections, in which case businesses are given 10 days to address those concerns. As the BOH is aware,these minor issues are almost always addressed within the 10 day period. The current process is clearly effective in addressing minor inspection issues. Our position is that the posting of all infractions online,however small,will garner a tremendous amount of negative feedback and unwarranted and unnecessary concerns-particularly on social media. On behalf of all Reading restaurants,we ask that you consider the significant unintended consequences of the proposed policy on our hardworking,locally owned businesses. We are fortunate to have a variety of amazing restaurants in our region,and even the whisper of a health infraction can dramatically affect a business and its viability.We would like to see the current policy remain,allowing time for the business to address the concerns,without causing the unnecessary alarm that an online public posting would cause: Respectfully, Francine Coughlin,Owner of Bark n' Roll and Chamber President Submitted on behalf of the Board of Directors - r MASSACHUSETTS DEPARTMENT OF PUBLIC HEAJL_T_H CBD (Cannabidiol) in Frrr Manufactured or e t in Massachusetts If 1 have a Massachusetts license or permit it legal to add hulled hemp seeds, hemp seed under 105 CMR 500 (Regulations for Good protein, and/or hemp seed oil to food I Manufacturing Practices for Food) to manufacture and/or sell at retail? manufacture food, is it legal add Yes. The FDA has completed its evaluation of cannabidiol (CBD) to that food? generally recognized as safe (GRAS) notices for No. The Massachusetts Department of Public hulled hemp seeds, hemp seed protein and Health (DPH) regulates food manufacturing in the hemp seed oil. These products can be legally Commonwealth (see 105 CMR 5001. These added to human food, provided they comply with regulations require that all food must be from all other requirements. approved sources that comply with federal, slate, and local law and must not contain any prohibited I have a license issued by the Massachusetts ingredients. The FDA has concluded that federal Department of Agricultural Resources (MDAR) law prohibits the addition of CBD to food under its Interim Commercial Industrial Hemp products because CBD is an active ingredient in Program Policy. Is it legal for me to add CBD FDA-approved drugs. Since CBD is not an to manufactured or retail food? approved ingredient under federal law, it may not be added to manufactured.foods. No. The MDAR Hemp Program does not provide an exception to the-felend prohibition against If I have a Massachusetts permit under 105 . adding CBD into food products. CMR 590(Minimum Sanitation Standards for May I market hem Food Establishments)as a food establishment, Y m Y p products (e.g., hulled is it legal to add CBD to food I distribute at hemp seeds, hemp seed protein and hemp retail? seed oil) by making therapeutic claims without FDA approval? No. The DPH sets minimum sanitary standards No.The FDA has reiterated that cannabis products for food establishments in the Commonwealth. These regulations are enforced by local boards (hemp-derived or otherwise)that are marketed with of health in partnership with the state (see 105 claims of therapeutic benefit or any other disease CM_ R 5901. The regulations incorporate the claim must be approved by the FDA. FDA's Food Code, which requires that food be May I seeti obtained from sources that comply with federalll cosmetic products containing hemp , or CBD ll retail Massachusetts? state, and local laws and must not contain any prohibited ingredients. As noted above, the FDA DPH does not regulate cosmetics. The FDA has .has concluded that federal law prohibits the issued Frequently Asked Questions that address addition of CBD to food products because CBD is cannabis and cannabis-derived ingredients in an active ingredient in FDA-approved drugs. cosmetics. Since CBD is not an approved ingredient under federal law, it may not be added to foods What can 1 do if I had an adverse reaction to a distributed at retail. cosmetic product labeled as containing_CBD? If I am licensed or permitted as a food You can report a cosmetic related complaint manufacturer or retailer in Massachusetts, is directly to the FDA. Massachusetts Department of Public Healthy y, Bureau of Environmental Health y4). 250 Washington Street, 7th Floor, Boston, MA 02108 V �tl1,t V Phone:617-624-57571 Fax:617-624-5777 1 TTY:617-624-5286 June 2019 w._ AAH MiyaresHarrington J.14 mtl Miyem Thomn l.4nnirgron CM1ri w,.,H.Heep Oenne M.Bnmr Jemia M.Menill PobeYeM1 teaay &pn Bartram Iv.ia 44n Feivd Plsundn O.PuhM KnAerine ESbd August 5, 2019 Board of Health 16 Lowell Street Reading, MA 01867 Re, Regulation of Cannabidiol (CBD) Products Dear Members of the Board of Health, On December 20, 2018, Congress legalized hemp and hemp-derived products. However, this does not mean that all products derived from hemp may be legally marketed and sold. Notably, CBD, or cannabidiol, is still prohibited when added to food products, and highly regulated when incorporated into other products. This letter will outline the authority of the Board of Health to regulate hemp and hemp-derived products sold in Town. Ma 'jua vs Hemp vs. CBD Marijuana and hemp are different varieties of the same plant species. Hemp is a non- psychoactive variety of the plant specifically cultivated for industrial uses. While marijuana plants contain high levels of delta 9-tetrahydrocannabinol (THC), the agent that gives marijuana its psychoactive attributes, hemp contains less than 0.3%THC on a dry-weight basis.' Cannabidiol(CBD)is derived from the cannabis plant and commonly extracted from hemp. As a hemp derivative, it is not considered to be marijuana. Overview of the Controlling Law Marijuana and hemp are also regulated differently. Marijuana is a Schedule 1 drug and illegal under the federal Controlled Subnance Aa. Massachusetts, like many states, has legalized marijuana for both adult(recreational) and medical use. The Cannabis Control Commission (CNB)is tasked with implementing and administering the laws enabling access to adult use and medical marijuana in the Commonwealth. M.G.L. c.94G and 935 CMR 500-502. At the local level, the Town of Reading prohibits all adult use marijuana establishments. The cultivation,production,and retail sale of Any product with more than 0.3%THC will be considered w be marijuana under federal and state law. 40Grove•Street1617459.i6M '.vo.«.miyares-hmrngmn cum. Local options at work Board of Health August S. 2019 Page 2 of 5 medical marijuana is allowed by special permit in certain zoning districts. Zoning Bylaw Section 5.3.1 and Section 5.6.5. In contrast, hemp is legal under both federal and state law. The U.S. Department of Agriculture (USDA) has been authorized to develop regulations and guidelines related to hemp.' The U.S. Food and Drug Administration (FDA)has also retained jurisdiction to address public health requirements for hemp-derived products. At the state level, the planting, growing,harvesting, processing, and sale of hemp or hemp products requires licensure by the Massachusetts Department of Agricultural Resources (MDAR). M.G.L. a 128, §§116— 123; Chapter 55 of the Acts of 2017. MDAR construes the terms"sell"or "sale of hemp to include the following sale transactions: • Massachusetts Licensed Grower to Massachusetts licensed Grower • Massachusetts Licensed Grower to Massachusetts Licensed Processor is Massachusetts Licensed Processor to Massachusetts retail facility(store) Thus, a license is not required for a retail facility to sell hemp or hemp-derived products to a consumer.' Like the FDA, the Massachusetts Department of Public Health (DPH)maintains authority for regulating food safety in the Commonwealth.' Recently, theFDAissued a statement that limits the types of hemp derived products that may be manufactured or sold to consumers.' Of relevance here is CBD. In its statement, the FDA affirmed that it is unlawful to manufacture or sell consumable products containing CBD or to 1 States are awaiting future direction and guidance from the USDA as to how to proceed. Until such time as Massachusetts receives additional information or a legislative change is made, the Massachusetts Department of Agricultural Resources has stated that it will continue to implement existing Massachusetts law. i Seg MDAR Policy Statement Regarding the Sale of Hemp-Derived Products in the Commonwealth(Jun. 12, 2019), attached as Exhibit A. The Town's Zoning Bylaw prohibits the cultivation,processing,and sale of"marijuana,"unless the use is permitted as a medical marijuana dispensary and licensed as such by the CNB.Zoning Bylaw Section 5.6.5.2.1."Marijuana"is defined in the bylaw to include hemp and hemp-derived products. Zoning Bylaw Section 2.0. Because a license from the CNB to cultivate,process,or sell hemp is not required(and cannot be obtained),such uses are unplicitly prohibited in the Town.The Community Planning and Development Commission is currently considering a proposed Zoning Bylaw amendment that would remove hemp and hemp-derived products from the definition of marijuana and thus permit the cultivation, processing,and sale of hemp and hemp-derived products within Town. s Statement from FDA Commissioner Scott Gottlieb, M.D.,on new steps to advance agency's continued evaluation of potential regulatory pathways for cannabis-containing and cannabis-derived products(Apr. 2, 2019),attached as Exhibit B;see also Statement From FDA Commissioner Scoot Gottlieb,M.D.,on signing of the Agriculture Improvement Art and the agency's regulation of products containing cannabis and carmabis-derived compounds(Dec.20, 2018), attached as Exhibit C;see also FDA Regulation of Cannabis and Cannabis-Derived Products,attached as Exhibit D. 40Groso Streat Suite 190 W.•:Isslev,Anassachuseus 02482 1 617.489.1600 1 wv,w.migenx-harringtonmm Local options at work Board of Health August 5, 2019 Page 3 of S market CBD as dietary supplements. The FDA justified the prohibition by pointing to the fact that "CBD is an active ingredient in FDA-approved drug products and were the subject of substantial clinical investigations before they were marketed as food."'The FDA also noted that"the only path that the [Federal Food, Drug, and Casuistic Actl allows for such substances to be added to foods or marketed as dietary supplements is if the FDA fust issues a regulation, through notice-and- comment rulemaking, allowing such use."M. In June, MDAR and the DPH adopted the FDA's guidance on the sale of hemp and hemp. derived products, including CBD Sale of Herrin and Hemp Derived Products In accordance with this guidance, the following hemp-derived products are approved for sale in the Commonwealth: - • Hemp seed; - • Hempseedoil; • .Hulled hemp; • Hemp seed powder; _ • Hemp protein; • Clothing; • Building material; • Items made from hemp fiber; and • Flower/plant from a Massachusetts licensed Grower to a Massachusetts licensed Grower or Processor. The following products are not approved for sale: • Any food product containing CBD; • Any product containing CBD derived from hemp that makes therapeutic/medicinal claims; • Any product that contains hemp as dietary supplement; • Animal feed that contains any hemp products; and $g4 Statement from FDA Commissioner Scott Gottlieb, M.D,Attachment B. 'Set MDAR Policy Statement(Jun. 12, 2019),attached as Exhibit A;see also DPH's Policy on CBD in Food Manufactured or Said in Massachusetts, attached as Exhibit E. 40 Grove Str e St ite 190 `/delleslcy, W--ch—nrm 02482 1 617:89.1600 1 www.mlyaro,haninytn�mm Local options at work Board of Health August 5, 2019 Page 4 of S is Unprocessed or raw plant material, including the flower that is meant for end use by a consumer.' Potential Stens As described above, adding CBD to food or other consumable products or selling food or other consumable products containing CBD violates federal and state law.v While we have yet to see any real movement to enforce this prohibition at the state level, certain municipalities, including North Reading, have issued warning letters to food establishments advising against the production, manufacturing, advertising, offering or selling of any food or other consumable products that contain CBD.10 Section 9 of the Reading Board of Health's regulations incorporates the State Sanitary Code Chapter X—Minimum Sanitation Standards for Food Establishments(105 CMR S90), which adopts the 2013 Federal Food Code, as amended. Thus, the Board of Health is empowered to bring an enforcement action against a food establishment that is selling food or other consumable products containing CBD. Section 1_8 of the Town's General Bylaw imposes a fine of$50 for the first offense, E 100 for the second offense, and E 150 for each additional offense." The Board should consider how it would like to approach the sale of food products and other consumables containing CBD within the Town and whether to notify food establishments of - the FDA and DPH's guidance prior to any enforcement action. °On June 28,2019,Representative Mark Cusack filed a bill in the Massachusetts House of Representatives(HD 4330)that would effectively reverse the guidelines from MDAR and the DPH.The bill would declare that hemp- derived carmabinoids,including CBD,are not controlled substances and therefore allowed be sold with oversight from the state. The bill would also declare that products containing CBD`intended for ingestion are to be considered foods,not controlled substances or adulterated products." °The FDA is starting to bring enforcement actions against manufactures of CBD products,especially those manufactures that make unsubstantiated claims that their products treat cancer,Alzheimer's disease,opioid withdrawal,pain,or pet anxiety.$yg,https://www.fda.gov/news-events/press-announcements/fda-warns- company-marketing-unapproved-cannabidiol-products-unsubstantiated-claims-treat-cancer. Certain states,like New York,have also aggressively begun enforcing the prohibition against the addition of CBD to food products,including coffee. S,&—e https://www.amny.com/eat-and-drink/nyc-cbd-food-ban-1.33245187. "a North Reading Board of Health,Mandatory Removal of All CBD Food Products(June 24, 2019),attached as Exhibit E "The Board of Health's regulations impose a fine of$25 for each violation,except for"critical"violations,which are punishable by 150,per offense. The Board may wish to consider amending its regulations to come into conformity with the Bylaw. 40 Grove Rreet Sulw190 - Weilesley, Msscachusotts 02482 1612.489.M001 nevm.,niyaras-harrington.com Local option at work Board of Health August 5, 2019 Page 5 of 5 l,ia Glass Fried, an attorney in my office, will be present at the Board's meeting on August 19. Please feel free to reach out before that time with any questions. Sincerely, J. Raymond Miyares cc: R. LeLacheur, Jr. Encl. 40 024521617.484.16001 wwwm 1-1n1-h_111ingmn com Local options at work EXHIBIT A THE COMMONWEALTH OF MASSACHUSETTS EXECUTIVE OFFICE OF ENERGY AND ENVIRONMENTAL AFFAIRS Department of Agricultural Resources 251 Causeway Street,Suite 500,Boston,MA 02114 ,MDAR 617-626-1700 fax: 617fiwaz I PF 26-1850 ww.ms.gov/agr aD�rN CHARLES D.BAKER KARYN E.POLITO KATHLEEN A.THEOHARIDES JOHN LEBEAUX Commissioner Governor L[,Governor Secretary POLICY STATEMENT REGARDING THE SALE OF HEMP-DERIVED PRODUCTS IN THE COMMONWEALTH This document sets forth the Massachusetts Department of Agricultural Resources("Department")Policy Statement on the Sale of Hemp-Derived Products in the Commonwealth("Policy').This Policy provides notice of categories of hemp-derived products that are approved for sale in the Commonwealth and identifies products that are prohibited for sale. This Policy also provides information on how a person or entity may become licensed ' to sell as required by M.G.L.c. 128,Section 118.All proposed activities related to the sale of hemp-derived v products must comply with this Policy in order to be considered in compliance with M.G.L.c. 128,Section 118. Please note that this Policy applies only to hemp and hemp-derived-products.All activities related to marijuana or the sale of any product containing marijuana are under the jurisdiction of the Cannabis Control Commission or law enforcement. CONTROLLING LAW The cultivation,processing,and sale of hemp in the Commonwealth of Massachusetts are regulated by both federal and state law. With the passing of the Agricultural Improvement Act of 2018,referred to as the"2018 Farm Bill,"signed on December 20, 2018,federal law now treats hemp as an agricultural commodity.Previously, hemp was not distinguished from marijuana under federal law,except under limited circumstances. Effective with this change in law, hemp is now legally recognized as a separate crop,the cultivation of which falls under the jurisdiction of the Unites States Department of Agriculture("USDA")and state departments of agriculture,if approved by USDA. Additionally,the United States Food and Drug Administration("FDA") retained its jurisdiction to address public health requirements for hemp-derived products under the Federal Food,Drug,and Cosmetic Act("FFDCA")and other related laws. The FDA has recently issued a statement that limits the types of cannabis-derived(including hemp-derived) products that may be manufactured and sold to consumers. In particular,the FDA has prohibited any food or other consumable products containing the cannabinoid known as"CBD"from interstate commerce without its approval.` Under state law,Sections 116 through 123 of Massachusetts General Laws Chapter 128 assign the Department responsibility to regulate all activities related to hemp and industrial hemp within the Commonwealth of Massachusetts.The law directs the Department to administer a licensing and registration program for the cultivation, processing,and sale of hemp and industrial hemp in the Commonwealth. More particularly,the law assigns the Department general oversight responsibility for approving the sale of hemp-derived products for 'See htt s: ww .fda. ov news-events rets-announcements statement-fda-commissioner-scott- ottlieb-md-new-steps - d -aR c stontinued-evaluation. commercial purposes within the Commonwealth.The Department's regulation of hemp-derived products is separate and in addition to controlling federal law, including FDA directives. The Massachusetts Department of Public Health("DPH")is responsible for regulating food safety in the Commonwealth. DPH has recently issued policy guidance that is consistent with the FDA policy in prohibiting the manufacture or sale of any food or other consumable products containing CBD.2 Under the state health and sanitary code, local boards of health have authority to enforce public health laws and regulations within a municipality. SALE OF HEMP-DERIVED PRODUCTS Under M.G.L. c. 128,Section 118,a license is required to"sell" industrial hemp(i.e.the products made from hemp).MDAR construes the term "sell"or"sale"of hemp to include the following sale transactions: • Massachusetts Licensed Grower to Massachusetts Licensed Grower • Massachusetts Licensed Grower to Massachusetts Licensed Processor • Massachusetts Licensed Processor to Massachusetts retail facility(store) Effective immediately,a person or entity that proposes to engage in the activities listed above shall be deemed licensed under M.G.L.c. 128,Section 118,provided that the person or entity is selling the hemp-derived products that are allowed under this Policy and is otherwise licensed as a Grower or Processor and in good standing with the Department.Such products are also deemed approved commercial uses under M.G.L.c. 128,Section 117(c). No further action or approval by the Department will be needed. All persons or entities proposing to sell hemp-derived products in the Commonwealth are also responsible for complying with all applicable federal and state laws and regulations, including any regulations or guidance issued by the FDA, DPH,or local boards of health.Applicable laws and regulations are as follows: Products for sale must be in compliance with the FDCA,as amended,and Section 351 of the Public Health Service Act. • Products must be in compliance with applicable DPH guidance. • Products must be allowed for sale under this Policy and,as such, be deemed to be approved commercial uses pursuant to M.G.L.c. 128,Section 117(c). APPROVED HEMP-DERIVED PRODUCTS The following hemp-derived products are approved for sale in the Commonwealth pursuant to M.G.L.c. 128, Section 117(c) and under FDA and DPH guidance: • Hemp seed • Hemp seed oil • Hulled hemp • Hemp seed powder • Hemp protein • Clothing • Building material • Items made from hemp fiber • Flower/plant from a Massachusetts licensed Grower to a Massachusetts licensed Grower or Processor 'See: httpi://www.mass-gov/info-detailsLcbd-in-food-manufactured-or-so)d-i - h tt Page 2 of 3 HEMP PRODUCTS NOT APPROVED FOR SALE The following products are NOT approved for sale in the Commonwealth pursuant to M.G.L.c. 128,Section 117(c) and are likewise prohibited for sale under FDA and DPH guidance: • Any food product containing CBO; • Any product containing CBD derived from hemp that makes the claims; • Any product that contains hemp as dietary supplement; • Animal feed that contains any hemp products;' • Unprocessed or raw plant material, including the flower that is meant for end use by a consumer. The Department will provide compliance assistance to individuals licensed pursuant to M.G. L.c. 128 who are interested in engaging in the sale of any hemp-derived products. However, it is the responsibility of any person involved with the sale of hemp-derived products to review and understand M.G.L.c. 128, Sections 116 through 123 and this Policy. Failure to comply may result in enforcement action from the Department, DPH,the local board of health,or law enforcement.This may include the inability to sell any hemp-derived products in the future,the destruction or seizure of illegal products, loss of license,or further legal action to determine whether the product sold is in compliance. 'See htts 'hvww.aofco.or-!Portals/0/SiteContent/Announcements/Guidelines on Hem in Animal Food May 1 20M&. Page 3 of 3 EXHIBIT B I7/2912019 Statement from FDA Commissioner Sea,,Gottlieb.M V,on new saps to advance agency's continued evaluation of pntemlal regulatory,pahwaya for c._ FDA STATEMENT Statement from FDA Commissioner Scott Gottlieb, M.D., on new steps to advance agency's continued evaluation of potential regulatory pathways for cannabis-containing and cannabis-derived products For Immediate Release: April 02,2019 Statement From: In recent years, we've seen a growing interest in the development of therapies and other FDA- regulated consumer products derived from cannabis(Cannabis sativa I..)and its components, including cannabidiol (CBD).This interest spans the range of product categories that the agency regulates. For example, we've seen, or heard of interest in,products containing cannabis or cannabis derivatives that are marketed as human drugs,dietary supplements,conventional foods, animal foods and drugs, and cosmetics,among other things.We also recognize that stakeholders are looking to the FDA for clarity on how our authorities apply to such products, what pathways are available to market such products lawfully under these authorities,and how the FDA is carrying out its responsibility to protect public health and safety with respect to such products. Interest in these products increased last December when Congress passed the Agriculture - Improvement Act of 2018 (the 2o18 Farm Bill).Among other things, this law established a new category of cannabis classified as"hemp" —defined as cannabis and cannabis derivatives with extremely low(no more than 0.3 percent on a dry weight basis)concentrations of the psychoactive compound delta-9-tetrahydrocannabinol(THC).The 2018 Farm Bill removed hemp from the Controlled Substances Act,which means that it is no longer a controlled substance under federal law. At the same time, Congress explicitly preserved the FDA's current authority to regulate products containing cannabis or cannabis-derived compounds under the Federal Food, Drug, and Cosmetic Act(FD&C Act) and section 351 of the Public Health Service Act. In doing so, Congress recognized the agency's important public health role with respect to all the products it regulates. This allows the FDA to continue enforcing the law to protect patients and the public while also providing potential regulatory pathways, to the extent permitted by law,for products containing cannabis and cannabis-derived compounds. hnprllwww rd.gaaln.ualeven¢/pressnnouncermnulslatemem-fda-ecmmuca.tier-soon-Baulieb-nal-news¢vs-advance-agency—onanued-evaluation 116 7/29/2019 Statementfrom FDA Commissioner Scoll Gottlieb.M.D..on new steps to advance agency's continued evaluation of ,memial regulamry pathways for a.. When the 2018 Farm Bill became law, I issued a statement(/news-events/press- a n nou nmments/statement-fda-commissioner-scoff-gottlieb-md-signing-agricultu re- improvement-act-and-ageneys)explaining the FDA's current approach to these products and our intended next steps. Consistent with the approach and commitments described in that statement, today the FDA is announcing a number of important new steps and actions to advance our consideration of a framework for the lawful marketing of appropriate cannabis and cannabis-derived products.under our existing authorities.These new steps include: • A public hearing(https://www.federalregister.gov/documents/2o19/04/03/2019- 06436/scientific-data-and-information-about-products-containing-cannabis-or-cannabis- derived-compounds)on May 31, as well as a broader opportunity for written public comment, for stakeholders to share their experiences and challenges with these products, including information and views related to product safety. • The formation of a high-level internal agency working group to explore potential pathways for dietary supplements and/or conventional foods containing CBD to be lawfully marketed; including a consideration of what statutory or regulatory changes might be needed and what the impact of such marketing would be on the public health. • Updates to our webpage (FDA Regulation of Cannabis and Cannabis-Derived Products: Questions and Answers) with answers to frequently asked questions on this topic to help members of the public understand how the FDA's requirements apply to these products. • The issuance of multiple warning letters to companies marketing CBD products with egregious and unfounded claims that are aimed at vulnerable populations. Public Hearing The public hearing will give stakeholders an opportunity to provide the FDA with additional input relevant to the agency's regulatory strategy related to existing products, as well as the lawful pathways by which appropriate products containing cannabis or cannabis-derived compounds can be marketed, and how we can make these legal pathways more predictable and efficient. We hope to gain additional information and data for the FDA to consider with respect to products containing cannabis and cannabis-derived compounds,including CBD. As we've stated before,we treat products containing cannabis or cannabis-derived compounds as we do any other FDA-regulated products.Among other things, the FDA requires a cannabis product(hemp-derived or otherwise) that's marketed with a claim of therapeutic benefit to be approved by the FDA for its intended use before it may be introduced into interstate commerce. Additionally, it is unlawful to introduce food containing added CBD, or the psychoactive compound THC, into interstate commerce,or to market CBD or THC products as dietary supplements.This is because CBD and THC are active ingredients in FDA-approved drug products and were the subject of substantial clinical investigations before they were marketed as food. In such situations,with certain exceptions that are not applicable here, the only path that mys:nwwwfda.gov/news-e.emr/prea:-anon„acemems/ammmem-faaKomm,astnneracmt-goober-md-new-steps-advanc -agencys<nmmnm-e.alua n ve 7892019 Summcnr from FDA Commissioner Scon Gmaicb.M.D..on new ir.,widi—e agency's continued evaluaion of poanual regula1ory paibwayz for c.. the FD&C Act allows for such substances to be added to foods or marketed as dietary supplements is if the FDA first issues a regulation, through notice-and-comment rulemaking, allowing such use. While the availability of CBD products in particular has increased dramatically in recent years, open questions remain regarding the safety considerations raised by their widespread use.For example, during its review of the marketing application for Epidiolex(/news-events/press- announcements/fda-approves-first-drug-comprised-active-ingredient-derived-marijuana-treat- rare-severe-forms) -a purified form of CBD that the FDA approved in 2018 for use in the treatment of certain seizure disorders - the FDA identified certain safety risks, including the potential for liver injury.These are serious risks that can be managed when the product is taken under medical supervision in accordance with the FDA-approved labeling for the product,but it is less clear how this risk might be managed in a setting where this drug substance is used far more widely,without medical supervision and not in accordance with FDA-approved labeling. There are also unresolved questions regarding the cumulative exposure to CBD if people access it across a broad range of consumer products, as well as questions regarding the intended functionality of CBD in such products.Additionally, there are open questions about whether some threshold level of CBD could be allowed in foods without undermining the drug approval process or diminishing commercial incentives for further clinical study of the relevant drug substance. It's critical that we address these unanswered questions about CBD and other cannabis and cannabis-derived products to help inform the FDA's regulatory oversight of these products- especially as the agency considers whether it could be appropriate to exercise its authority to allow the use of CBD in dietary supplements and other foods.As I stated in December, the FDA would only consider this path if the agency were able to determine that all other requirements in the FD&C Act are met,including those required for food additives or new dietary ingredients. As part of the public hearing and related public comment period,the agency is interested in whether there are particular safety concerns that we should be aware of as we consider the FDA's regulatory oversight and monitoring of these products. For example,we're seeking comments, data and information on a variety of topics including: what levels of cannabis and cannabis- derived compounds cause Aafety concerns;how the mode of delivery(e.g.,ingestion, absorption, - inhalation) affects the safety of,and exposure to,these compounds;how cannabis and cannabis- derived compounds interact with other substances such as drug ingredients; and other questions outlined in the hearing announcement. Additionally, we're interested in how the incentives for,and the feasibility of, drug development with CBD and other cannabis-derived compounds would be affected if the commercial availability of products with these compounds,such as foods and dietary supplements,were to become significantly more widespread.We don't want companies to forgo research that might support approval through the FDA's drug review process, which could potentially lead to ,�-goareb-nm-new steps-aavanc a ncra< ntmuea-evaluauon vu nuva:owww•_ma.ewlnew,-aenwvmaa annuun«mena/auamem-faa�nmmaa,one.aw ° 7n9no 19 Smtemem from FDA Commissioner Scon Go1111cb,M.D.,net new steps to advance agency's continued evaluation of potential regulatory pathways for c.._ important safe and effective therapies. We also don't want patients to forgo appropriate medical treatment by substituting unapproved products for approved medicines used to prevent, treat, mitigate or cure a particular disease or condition. For example, in the case of Epidiolex, the adequate and well-controlled clinical studies that supported its approval, and the assurance of manufacturing quality standards, can provide prescribers confidence in the drug's uniform strength and consistent delivery that support appropriate dosing needed for treating patients with these complex and serious epilepsy syndromes. It's important that we continue to assess whether there could be medical ramifications if patients choose to take CBD to treat certain diseases at levels higher or lower than studied in well-controlled clinical studies. FDA Working Group We hope that information we receive through the public hearing this May, as well as through the written public comment process, will help inform our consideration of these and other - important scientific,technical and policy questions. Given the importance of these questions, and the significant public interest with respect to CBD in particular, we're forming a high-level internal agency working group to explore potential pathways for dietary supplements and/or conventional foods containing CBD to be lawfully marketed.Given the importance of this issue, I've asked Principal Deputy Commissioner Amy Abernethy, M.D., Ph.D. and Principal Associate Commissioner for Policy Lowell Schiller,to co-chair the group and charged them with considering what options might be appropriate under our current authorities, in view of all the evidence before us and our agency's fundamental public health mission. I'm also asking the group to consider whether there are legislative options that might lead to more efficient and appropriate pathways than might be available under current law—again,with the same science- based,public health focus that the FDA endeavors to bring to all matters before it.This is a. - complicated topic and we expect that it could take some time to resolve fully. Nevertheless,we're deeply focused on this issue and committed to continuing to engage relevant stakeholders as we consider potential paths forward.The working group plans to begin sharing information and/or findings with the public as early as Summer 2019. New Comp(iunce Actions We'll continue to use our authorities to take action against companies illegally selling.these types of products when they are putting consumers at risk. I am deeply concerned about any circumstance where product developers make unproven claims to treat serious or life- threatening diseases, and where patients may be misled to forgo otherwise effective,available therapy and opt instead for a product that has no proven value or may cause them serious harm. Today, the FDA is announcing that it has issued warning letters,in collaboration with the Federal Trade Commission, to three companies—Advanced Spine and Pain LLC(d/b/a Relievus) (Advanced Spine and Pain, LLC(d/b/a Relievus)), Nutra Pure LLC(/inspections- compliance-enforcement-and-cri urinal-investigations/warning-letters/nutra-pure-Ile-B 6yyt4- hdP.nwww.fda gov(newse cnt0pre a-announcements'smenem-fda-commissioner-scon-gou,m-and-new steps ad,ane -agenga-continued-evaluation 416 M912019 Saanmm Gom MA M D..on new smpnoa&.—agency's ominue0 cvalua�ion of p"mmial rtgulesory pashwnys for e.. 03282019) and PotNetwork Holdings Inc. (/inspections-compliance-enforcement-and-criminal- investigations/warning-letters/potnetwork-holdings-inc-564030-03282019) — in response to their making unsubstantiated claims related to more than a dozen different products and spanning multiple product webpages, online stores and social media websites.The companies used these online platforms to make unfounded, egregious claims about their products' ability to limit, treat or cure cancer, neurodegenerative conditions, autoimmune diseases, opioid use disorder, and other serious diseases, without sufficient evidence and the legally required FDA approval. Examples of claims made by these companies include: • "CBD successfully stopped cancer cells in multiple different cervical cancer varieties." • "CBD also decreased human glioma cell growth and invasion,thus suggesting a possible role of CBD as an antitumor agent." • "For Alzheimer's patients, CBD is one treatment option that is slowing the progression of that disease." • "Fibromyalgia is conceived as a central sensitization state with secondary hyperalgesia. CBD has demonstrated the ability to block spinal, peripheral and gastrointestinal mechanisms responsible for the pain associated with migraines, fibromyalgia, IBS and other related disorders." • "Cannabidiol May be Effective for Treating Substance Use Disorders." • "CBD reduced the rewarding effects of morphine and reduced drug seeking of heroin." • "CBD may be used to avoid or reduce withdrawal symptoms." I believe these are egregious,over-the-line claims and we won't tolerate this kind of deceptive marketing to vulnerable patients.The FDA continues to be concerned about the proliferation of egregious medical claims being made about products asserting to contain CBD that haven't been approved by the FDA,such as the products and companies receiving warning letters today. CBD is marketed in a variety of product types,such as oil drops,capsules, syrups, teas and topical lotions and creams. Often such products are sold online and are therefore available throughout the country. Selling unapproved products with unsubstantiated therapeutic claims can put patients and consumers at risk.These products have not been shown to be safe or effective, and deceptive marketing of unproven treatments may keep some patients from accessing appropriate, recognized therapies to treat serious and even fatal diseases.Additionally, because they are not evaluated by the FDA, there may be other ingredients that are not disclosed, which may be harmful. As our actions today make clear, the FDA stands ready to protect consumers from companies illegally selling CBD products that claim to prevent, diagnose, treat, or cure serious diseases, such as cancer, Alzheimer'sdisease, psychiatric disorders and diabetes.The agency has and will nvpa:I/—finen.lnews<.emsrpress-announcemen✓sulemrnl-raa<ommissiener-scot-gnfflieb-me-newaaps-aa.ante-agencys.cenlinuea-evamaeon s16 7/29/2019 Smromemfran FDA Commivioner Scat Gottlieb,M.D..on new steps to advance agency's continued evaluation of oten ial«gulatory porways for c,.. continue to monitor the marketplace and take enforcement action as needed to protect the public health against companies illegally selling cannabis and cannabis-derived products that can put consumers at risk and are being marketed and distributed in violation of the FDA's authorities. Ultimately, we remain committed to exploring an appropriate,efficient and predictable regulatory framework to allow product developers that meet the requirements under our authorities to lawfully market these types of products. The actions we're announcing today will allow us to continue to clarify our regulatory authority over these products and seek input from a broad range of stakeholders and examine a variety of approaches and considerations in the marketing and regulation of cannabis or cannabis-derived products, while continuing to protect the public's health and safety. The FDA, an agency within the U.S. Department of Health and Human Services,protects the public health by assuring the safety, effectiveness,and security of human and veterinary drugs, vaccines and other biological products for human use,and medical devices. The agency also is responsible for the safety and security of our nation's food supply, cosmetics,dietary supplements, products that give off electronic radiation, and for regulating tobacco products. it y uirics Media: M Michael Felberbaum (mailto:michael.felberbaum(la fda.hhs.gov) L 240-402-9548 Consumer: t 888-INFO-FDA ©More Press Announcements(/news-events/newsroom/press-announcements) hops IIwwwHagar/nees—ems/press-announeeseents/su¢ment-fda-commissioner-stun-goulieQmtl-newnepsadvane encys-eonunued-oval uauon 616 EXHIBIT C 112112019 Statement from FDA Commissionef Scau Gottlieb,MD..on signing of the Agdculmre lmprou,cment Act and the agency's regulation of products coma... FDA STATEMENT Statement from FDA Commissioner Scott Gottlieb, M.D., on signing of the Agriculture Improvement Act and the agency's regulation of products containing cannabis and cannabis- derived compounds For Immediate Release: December 20, 2018 Statement From: - Commissioner of Food and Drugs-Food and Drug Administration Scott Gottlieb M.D. Today, the Agriculture Improvement Act of 2018 was signed into law.Among other things,this new law changes certain federal authorities relating to the production and marketing of hemp, defined as cannabis (Cannabis satiaa L.),and derivatives of cannabis with extremely low(less than 0.3 percent on a dry weight basis)concentrations of the psychoactive compound delta-q- tetrahydrocannabinol (THC).These changes include removing hemp from the Controlled Substances Act,which means that it will no longer be an illegal substance under federal law. Just as important for the FDA and our commitment to protect and promote the public health is what the law didn't change: Congress explicitly preserved the agency's current authority to regulate products containing cannabis or cannabis-derived compounds under the Federal Food, Drug,and Cosmetic Act(FD&C Act) and section 351 of the Public Health Service Act. In doing so, Congress recognized the agency's important public health role with respect to all the products it regulates.This allows the FDA to continue enforcing the law to protect patients and the public while also providing potential regulatory pathways for products containing cannabis and cannabis-derived compounds. We're aware of the growing public interest in cannabis and cannabis-derived products, including cannabidiol (CBD).This increasing public interest in these products makes it even more important with the passage of this law for the FDA to clarify its regulatory authority over these products. In short,we treat products containing cannabis or cannabis-derived compounds as we do any other FDA-regulated products— meaning they're subject to the same authorities and requirements as FDA-regulated products containing any other substance. This is true regardless of the source of the substance, including whether the substance is derived from a plant that is classified as hemp under the Agriculture Improvement Act.To help members of the public understand how the FDA's requirements apply to these products, the FDA has maintained a Impz://www Ida govtnews-cents/press-announcememslsu¢mem-fdasommissionervscoo-gonGeb-md-signingagdculmremproementacoand-ageneys 114 71232019 Statement from FDA Commissioner Seem Gottlieb.M.D.,on signing of the Agneultme Improvement Act and the agency's regulmion of products conn... webpage(/news-events/public-health-focus/fda-regulation-cannabis-and-cannabis-derived- products-questions-and-answers)with answers to frequently asked questions,which we intend to update moving forward to address questions regarding the Agriculture Improvement Act and regulation of these products generally. In view of the proliferation of products containing cannabis or cannabis-derived substances,the FDA will advance new steps to better define our public health obligations in this area.We'll also continue to closely scrutinize products that could pose risks to consumers.Where we believe consumers are being put at risk, the FDA will warn consumers and take enforcement actions. In particular,we continue to be concerned at the number of drug claims being made about products not approved by the FDA that claim to contain CBD or other cannabis-derived compounds. Among other things,the FDA requires a cannabis product(hemp-derived or otherwise) that is marketed with a claim of therapeutic benefit, or with any other disease claim, to be approved by the FDA for its intended use before it may be introduced into interstate commerce. This is the same standard to which we hold any product marketed as a drug for human or animal use.Cannabis and cannabis-derived products claiming in their marketing and promotional materials that they're intended for use in the diagnosis, cure, mitigation, treatment, or prevention of diseases(such as cancer,Alzheimer's disease, psychiatric disorders and diabetes) are considered new drugs or new animal drugs and must go through the FDA drug approval process for human or animal use before they are marketed in the U.S.Selling unapproved products with unsubstantiated therapeutic claims is not only a violation of the law, but also can put patients at risk,as these products have not been proven to be safe or effective. This deceptive marketing of unproven treatments raises significant public health concerns, as it may keep some patients from accessing appropriate, recognized therapies to treat serious and even fatal diseases. Additionally, it's unlawful under the FD&C Act to introduce food containing added CBD or THC into interstate commerce,or to market CBD or THC products as, or in, dietary supplements, regardless of whether the substances are hemp-derived.This is because both CBD and THC are active ingredients in FDA-approved drugs and were the subject of substantial clinical investigations before they were marketed as foods or dietary supplements. Under the FD&C Act, it's illegal to introduce drug ingredients like these into the food supply, or to market them as dietary supplements.This is a requirement that we apply across the board to food products that contain substances that are active ingredients in any drug. We'll take enforcement action needed to protect public health against companies illegally selling cannabis and cannabis-derived products that can put consumers at risk and are being marketed in violation of the FDA's authorities.The FDA has sent warning letters(/news-events/public- health-focus/warning-letters-and-test-results-cannabidiol-related-products) in the past to companies illegally selling CBD products that claimed to prevent,diagnose, treat, or cure serious nnpsnwwv.Mat gnvmea-ev<nmpres:annonnceme nrsrsmtement-fdatommissmner-smn-gopllen-md.signing-agoi ulmre-Impro ement-aa-and-agencys gra 7M20[9 Sm¢mcm from MA C'nadn"oncr Scon 0.11646,M.D.,on si 8-3-9 Of da A,­n1,.n,InIPOO'emend Acl and Conia diseases,such as cancer. Some of these products were in further violation of the FD&C Act p,ndncaa because they were marketed as dietary supplements or because they involved the addition of CBD to food. While products containing cannabis and cannabis-derived compounds remain subject to the FDA's authorities and requirements, there are pathways available for those who seek to lawfully introduce these products into interstate commerce. The FDA will continue to take steps to make the pathways for the lawful marketing of these products more efficient. These pathways include ways for companies to seek approval from the FDA to market with therapeutic claims a human or animal drug that is derived from cannabis. For example, in June 2018, the FDA approved a drug, Epidiolex(/news-events/press-announcements/fda-approves- first-drug-comprised-active-ingredient-derived-marijuana-treat-rare-severe-forms),that contains cannabis-derived CBD for the treatment of seizures associated with two rare and severe forms of epilepsy. That approval was based on adequate and well-controlled clinical studies, which gives prescribers confidence in the drug's uniform strength and consistent delivery that support appropriate dosing needed for treating patients with these complex and serious epilepsy syndromes. In addition, pathways remain available for the FDA to consider whether there are circumstances in which certain cannabis-derived compounds might be permitted in a food or dietary supplement.Although such products are generally prohibited to be introduced in interstate commerce, the FDA has authority to issue a regulation allowing the use of a pharmaceutical ingredient in a food or dietary supplement.We are taking new steps to evaluate whether we should pursue such a process. However,the FDA would only consider doing so if the agency were able to determine that all other requirements in the FD&C Act are met, including those required for food additives or new dietary ingredients. It should also be noted that some foods are derived from parts of the hemp plant that may not contain CBD or THC, meaning that their addition to foods might not raise apse the same issues as the addition of drug ingredients like CBD and THC.We are able to advance the lawful marketing of three such ingredients today.We are announcing that the agency has completed our evaluation of three Generally Recognized as Safe(/food/cfsan-constituent-updates/fda-responds-three- gras-notices-hemp-seed-derived-ingredients-use-human-food) (GRAS) notices related to hulled hemp seeds, hemp seed protein and hemp seed oil and that the agency had no questions regarding the company's conclusion that the use of such products asdescribedin the notices is safe. Therefore,these products can be legally marketed in human foods for these uses without food additive approval, provided they comply with all other requirements and do not make disease treatment claims. hugs Hwwwfda,00mews e,enaipr<sa annnuncement0satement-fda<ommisszoner-:cm,-gotnleb-mtlsigningagiiculw m m mpi,in-1 cl-and agency+ 314 72112019 Statement from FDA Commissioner Scott Goodies.MD..on signing of the Agneahme lmproement Act and Ne agency's regulation of producumnu.. Given the substantial public interest in this topic and the clear interest of Congress in fostering the development of appropriate hemp products,we intend to hold a public meeting in the near future for stakeholders to share their experiences and challenges with these products, including information and views related to the safety of such products. We'll use this meeting to gather additional input relevant to the lawful pathways by which products containing cannabis or cannabis-derived compounds can be marketed,and how we can make these legal pathways more predictable and efficient.We'll also solicit input relevant to our regulatory strategy related to existing products,while we continue to evaluate and take action against products that are being unlawfully marketed and create risks for consumers. At the same time, we recognize the potential opportunities that cannabis or cannabis-derived compounds could offer and acknowledge the significant interest in these possibilities.We're committed to pursuing an efficient regulatory framework for allowing product developers that meet the requirements under our authorities to lawfully market these types of products. The FDA, an agency within the U.S. Department of Health and Human Services,protects the public health by assuring the safety,effectiveness, and security of human and veterinary drugs, vaccines and other biological products for human use, and medical devices.The agency also is responsible for the safety and security of our nation's food supply,cosmetics, dietary supplements,products that give off electronic radiation, and for regulating tobacco products. Inquiries Media: u Lyndsay Meyer(mailto:lyndsay.meyer@fda.hhs.gov) L 240-402-5345 Consumer: t 888-INFO-FDA ©More Press Announcements(/news-events/newsroom/press-announcements) nopr:nwwr,..fdaon.mew:-e. nulpres.-announamems'statement-fda.eommissionersenu-gotaien.md.signig-agnemmre-impro ement-act.and-asencys a4 EXHIBIT D FDA Regulation of Cannabis and Cannabis-Derived Products: Questions and Answers On this page: • Latest News • Questions and Answers • Related Information Over the past decade, there has been a growing interest in the development of therapies and other consumer products derived from cannabis and its components, including cannabidiol (CBD). FDA recognizes the potential opportunities that cannabis or cannabis-derived compounds may offer and acknowledges the significant interest in these possibilities. However, FDA is aware that some companies are marketing products containing cannabis and cannabis- derived compounds in ways that violate the Federal Food, Drug and Cosmetic Act(FD&C Act) and that may put the health and safety of consumers at risk.The agency is committed to protecting the public health while also taking steps to improve the efficiency of regulatory pathways for the lawful marketing of appropriate cannabis and cannabis-derived products. Latest News In order to inform the regulatory path forward, FDA held a public hearing(/news- events/fda-meetings-conferences-and-workshops/scientific-data-and- information-about-products-containing-cannabis-or-cannabis-derived- ! compounds) on May 31, 2019, for stakeholders to share their experiences and challenges with products containing cannabis and cannabis-derived compounds, including information and views related to the safety of such products,as well as to solicit input relevant to the agency's regulatory strategy related to existing products.As part of that hearing, FDA opened a docket for the public to submit comments (https://www.regulations.gov/docket?D=FDA-2oi9-N-1482)through July 16, 2019. Questions and Answers Below are a number of frequently asked questions and answers on this topic. 1. What are cannabis and marijuana? 2. How does the 2018 Farm Bill define hemp?What does it mean for FDA-regulated products? 3. Has FDA approved any medical products containing cannabis or cannabis-derived compounds such as CBD? 4.Aside from Epidiolex,are there other CBD drug products that are FDA-approved?What about the products I've seen in stores or online? 5. Why hasn't FDA approved more products containing cannabis or cannabis-derived compounds for medical uses? 6. What is FDA's reaction to states that are allowing cannabis to be sold for medical uses without the FDA's approval? 7. Has the agency received any adverse event reports associated with cannabis use for medical conditions? 8. Is it legal for me to sell CBD products? 9. Can THC or CBD products be sold as dietary supplements? 1o. Is it legal, in interstate commerce,to sell a food(including any animal food or feed)to which THC or CBD has been added? 11. In making the two previous determinations about THC,why did FDA conclude that THC is at..active ingredient in a drug product that has been approved under section 505 of the FD&C Act? In making the two previous determinations about CBD,why did FDA determine that substantial clinical investigations have been authorized for and/or _ instituted,and that the existence of such investigations has been made public? 12. Can hulled hemp seed,hemp seed protein powder,and hemp seed oil be used in human food? 13. What is FDA's position on cannabis and cannabis-derived ingredients in cosmetics? 14. Will FDA take action against cannabis or cannabis-related products that are in violation of the FD&C Act? 15. Can I import or export cannabis-containing or cannabis-derived products? 16. What is FDA's role when it comes to the investigation of cannabis and cannabis-derived products for medical use? 17. Does the FDA object to the clinical investigation of cannabis for medical use? 18. How can patients gain access to cannabis or cannabis-derived products for medical use through expanded access? in. Can patients gain access to cannabis or cannabis-derived products for medical use through Right to Try? 20. Does the FDA have concerns about administering a cannabis product to children? 21. Does the FDA have concerns about administering a cannabis product to pregnant and lactating women? 22. What does the FDA think about making CBD available to children with epilepsy? 23. What should I do if my child eats something containing cannabis? 24. I've seen cannabis products being marketed for pets.Are they safe? 25. Can hemp be added to animal food? - 26. Can approved human drugs containing CBD or synthetic THC be used extralabel in animals? 1. What are cannabis and marijuana? A. Cannabis is a plant of the Cannabaceae family and contains more than eighty biologically active chemical compounds.The most commonly known compounds are delta-9- tetrahydrocannabinol(THC)and cannabidiol(CBD). Parts of the Cannabis saliva plant have been controlled under the Controlled Substances Act(CSA)since 1970 under the drug class "Marihuana"(commonly referred to as"marijuana") [21 U.S.C. 802(16)). "Marihuana"is listed in Schedule I of the CSA due to its high potential for abuse,which is attributable in large part to the psychoactive effects of THC,and the absence of a currently accepted medical use of the plant in the United States. z. How does the z018 Farm Bill define hemp?What does it mean for FDA- regulated products? A. At the federal level,the Agriculture Improvement Act of 2oi8, Pub. L. 115-334, (the 2018 Farm Bill) was signed into law on Dec. 20, 2018.Among other things,this new law changes certain federal authorities relating to the production and marketing of hemp, defined as "the plant Cannabis sativa L. and any part of that plant, including the seeds thereof and all derivatives, extracts, cannabinoids, isomers,acids,salts,and salts of isomers,whether growing or not,with a delta-9 tetrahydrocannabinol concentration of not more than 0.3 percent on a dry weight basis."These changes include removing hemp from the CSA, which means that cannabis plants and derivatives that contain no more than 0.3 percent THC on a dry weight basis are no longer controlled substances under federal law. The 2018 Farm Bill, however, explicitly preserved FDA's authority to regulate products containing cannabis or cannabis-derived compounds under the FD&C Act and section 351 of the Public Health Service Act (PHS Act). FDA treats products containing cannabis or cannabis- derived compounds as it does any other FDA-regulated products— meaning they're subject to the same authorities and requirements as FDA-regulated products containing any other substance.This is true regardless of whether the cannabis or cannabis-derived compounds are classified as hemp under the 2018 Farm Bill. 3.Has FDA approved any medical products containing cannabis or cannabis- derived compounds such as CBD? A.To date, the agency has not approved a marketing application for cannabis for the treatment of any disease or condition. FDA has, however,approved one cannabis-derived and three cannabis-related drug products.These approved products are only available with a prescription from a licensed healthcare provider. FDA has approved Epidiolex,which contains a purified form of the drug substance CBD for the - treatment of seizures associated with Lennox-Gastaut syndrome or Dravet syndrome in patients z years of age and older.That means FDA has concluded that this particular drug product is safe and effective for its intended use. The agency also has approved Marinol and Syndros for therapeutic uses in the United States, including for the treatment of anorexia associated with weight loss in AIDS patients. Marinol and Syndros include the active ingredient dronabinol,a synthetic delta-q-tetrahydrocannabinol (THC)which is considered the psychoactive component of cannabis.Another FDA-approved drug,Cesamet,contains the active ingredient nabilone,which has a chemical structure similar to THC and is synthetically derived. 4.Aside from Epidiolex, are there other CBD drug products that are FDA- approved?What about the products I've seen in stores or online? A. No.There are no other FDA-approved drug products that contain CBD.We are aware that some firms are marketing CBD products to treat diseases or for other therapeutic uses , and we have issued several warning letters(/news-events/public-health-focus/warning-letters-and-test- results-cannabidiol-related-products)to such firms. Under the FD&C Act,any product intended to have a therapeutic or medical use,and any product(other than a food)that is intended to affect the structure or function of the body of humans or animals, is a drug. Drugs must generally either receive premarket approval by FDA through the New Drug Application(NDA) process or conform to a "monograph" for a particular drug category, as established by FDA's Over-the-Counter(OTC) Drug Review. CBDwasnot an ingredient considered under the OTC drug review. An unapproved new drug cannot be distributed or sold in interstate commerce. FDA continues to be concerned at the proliferation of products asserting to contain CBD that are marketed for therapeutic or medical uses although they have not been approved by FDA. Often such products are sold online and are therefore available throughout the country. Selling unapproved products with unsubstantiated therapeutic claims is not only a violation of the law, but also can put patients at risk,as these products have not been proven to be safe or effective. This deceptive marketing of unproven treatments also raises significant public health concerns, because patients and other consumers may be influenced not to use approved therapies to treat serious and even fatal diseases. Unlike drugs approved by FDA, products that have not been subject to FDA review as part of the drug approval process have not been evaluated as to whether they work, what the proper dosage may be if they do work,how they could interact with other drugs, or whether they have dangerous side effects or other safety concerns. The agency has and will continue to monitor the marketplace and take action as needed to protect the public health against companies illegally selling cannabis and cannabis-derived products that can put consumers at risk and that are being marketed for therapeutic uses for which they are not approved.At the same time, FDA.recognizes the potential therapeutic opportunities that cannabis or cannabis-derived compounds could offer and acknowledges the significant interest in these possibilities. FDA continues to believe that the drug approval process represents the best way to help ensure that safe and effective new medicines, including any drugs derived from cannabis,are available to patients in need of appropriate medical therapy. The Center for Drug Evaluation and Research(ODER) is committed to supporting the development of new drugs, including cannabis and cannabis-derived drugs, through the investigational new drug (IND)and drug approval process(see Question#t6). 5.Why hasn't FDA approved more products containing cannabis or cannabis- derived compounds for medical uses? A. FDA is aware that unapproved cannabis or cannabis-derived products are being used for the treatment of a number of medical conditions including, for example,AIDS wasting, epilepsy, neuropathic pain,spasticity associated with multiple sclerosis, and cancer and chemotherapy- induced nausea. To date, FDA has not approved a marketing application for cannabis for the treatment of any disease or condition and thus has not determined that cannabis is safe and effective for any particular disease or condition.The agency has, however,approved one cannabis-derived and three cannabis-related drug products(see Question #2). FDA relies on applicants and scientific investigators to conduct research.The agency's role, as laid out in the FD&C Act, is to review data submitted to the FDA in an application for approval to ensure that the drug product meets the statutory standards for approval. The study of cannabis and cannabis-derived compounds in clinical trial settings is needed to assess the safety and effectiveness of these substances for the treatment of any disease or condition. FDA's December 2oi6 Guidancefor Industry: Botanical Drug Development (/regulatory-information/search-fda-guidance-documents/botanical-drug-development- guidance-industry) provides specific recommendations on submitting INDs for botanical drug products,such as those derived from cannabis, in support of future marketing applications for these products. The FDA will continue to facilitate the work of companies interested in appropriately bringing safe,effective, and quality products to market,including scientifically- based research concerning the medicinal uses of cannabis.Additional information concerning research on the medical use of cannabis is available from the National Institutes of Health, particularly the National Cancer Institute (https://www.cancer.gov/) (NCI) and National Institute on Drug Abuse (https://www.drugabuse.gov/drugs-abuse/marijuana/nih-research- marijuana-cannabinoids) (NIDA). 6. What is FDA's reaction to states that are allowing cannabis to be sold for medical uses without the FDA's approval? A.The FDA is aware that several states have either passed laws that remove state restrictions on the medical use of cannabis and its derivatives or are considering doing so. It is important to conduct medical research into the safety and effectiveness of cannabis products through adequate and well-controlled clinical trials.We welcome the opportunity to talk with states who are considering support for medical research of cannabis and its derivatives,so that we can provide information on Federal and scientific standards. 7. Has the agency received any adverse event reports associated with cannabis use for medical conditions? - A.The agency has received reports of adverse events in patients using cannabis or cannabis- derived products to treat medical conditions.The FDA reviews such reports and will continue to monitor adverse event reports for any safety signals,with a focus on serious adverse effects. Information from adverse event reports regarding cannabis use is extremely limited; FDA primarily receives adverse event reports for approved products. General information on the potential adverse effects of using cannabis and its constituents can come from clinical trials that have been published, as well as from spontaneously reported adverse events sent to the FDA. Additional information about the safety and effectiveness of cannabis and its constituents is needed. Clinical trials of cannabis conducted under an IND application could collect this important information as a part of the drug development process. S. Is it legal for me to sell CBD products? A. It depends,among other things,on the intended use of the product and how it is labeled and marketed. Even if a CBD product meets the definition of"hemp" under the 2018 Farm Bill(see Question nz), it still must comply with all other applicable laws, including the FD&C Act.The below questions and answers explain some of the ways that specific parts of the FD&C Act can affect the legality of CBD products. We are aware that state and local authorities are fielding numerous questions about the legality of CBD.There is ongoing communication with state and local officials to answer questions about requirements under the FD&C Act, to better understand the landscape at the state level, and to otherwise engage with state/local regulatory partners. 9. Can THC or CBD products be sold as dietary supplements? A. No. Based on available evidence, FDA has concluded that THC and CBD products are excluded from the dietary supplement definition under section 201(ff)(3)(B) of the FD&C Act[21 U.S.C. §321(ff)(3)(B)]. Under that provision, if a substance(such as THC or CBD) is an active ingredient in a drug product that has been approved under section 505 of the FD&C Act[21 U.S.C. § 3553, or has been authorized for investigation as a new drug for which substantial clinical investigations have been instituted and for which the existence of such investigations has been made public, then products containing that substance are excluded from the definition of a dietary supplement. FDA considers a substance to be "authorized for investigation as a new drug"if it is the subject of an Investigational New Drug application(IND) that has gone into effect. Under FDA's regulations(21 CFR 312.2),unless a clinical investigation meets the limited criteria in that regulation, an IND is required for all clinical investigations of products that are subject to section 505 of the FD&C Act. There is an exception to section 201(ff)(3)(B) if the substance was "marketed as"a dietary supplement or as a conventional food before the drug was approved or before the new drug investigations were authorized,as applicable. However,based on available evidence, FDA has concluded that this is not the case for THC or CBD. FDA is not aware of any evidence that would call into question its current conclusions that THC and CBD products are excluded from the dietary supplement definition under section 201(ft)(3) (B) of the FD&C Act. Interested parties may present the agency with any evidence that they think has bearing on this issue. Our continuing review of information that has been submitted thus far has not caused us to change our conclusions. When a substance is excluded from the dietary supplement definition under section 201(ff)(3) (B) of the FD&C Act, the exclusion applies unless FDA,in the agency's discretion, has issued a regulation,after notice and comment,finding that the article would be lawful under the FD&C Act.To date, no such regulation has been issued for any substance. Ingredients that are derived from parts of the cannabis plant that do not contain THC or CBD might fall outside the scope of this exclusion, and therefore might be able to be marketed as dietary supplements. However,all products marketed as dietary supplements must comply with all applicable laws and regulations governing dietary supplement products. For example, manufacturers and distributors who wish to market dietary supplements that contain "new dietary ingredients" (i.e., dietary ingredients that were not marketed in the United States in a dietary supplement before October 15, 1994)generally must notify FDA about these ingredients (see section 413(d) of the FD&C Act [21 U.S.C. § 35ob(d)]). Generally,the notification must include information demonstrating that a dietary supplement containing the new dietary ingredient will reasonably be expected to be safe under the conditions of use recommended or suggested in the labeling. A dietary supplement is adulterated if it contains a new dietary ingredient for which there is inadequate information to provide reasonable assurance that the ingredient does not present a significant or unreasonable risk of illness or injury(see section 402(f)(1)(B) of the FD&C Act[21 U.S.C.342(f)(1)(B)]). Numerous other legal requirements apply to dietary supplement products, including requirements relating to Current Good Manufacturing Practices(CGMPs) (/food/current-good- manufacturing-practices-cgmps/current-good-manufacturing-practices-cgmps-dietary- supplements) and labeling. Information about these requirements, and about FDA requirements across all product areas, can be found on FDA's website. 10. Is it legal, in interstate commerce, to sell a food(including any animal food or feed) to which THC or CBD has been added? - - A. No. Under section 301(11) of the FD&C Act[21 U.S.C. § 33101)], it is prohibited to introduce or deliver for introduction into interstate commerce any food(including any animal food or feed) to which has been added a substance which is an active ingredient in a drug product that has been approved under section 505 of the FD&C Act [21 U.S.C. § 3551,or a drug for which substantial clinical investigations have been instituted and for which the existence of such investigations has been made public.There are exceptions,including when the drug was marketed in food before the drug was approved or before the substantial clinical investigations involving the drug had been instituted or, in the case of animal feed,that the drug is a new animal drug approved for use in feed and used according to the approved labeling.However,based on available evidence, FDA has concluded that none of these is the case for THC or CBD. FDA has therefore concluded that it is a prohibited act to introduce or deliver for introduction into interstate commerce any food (including any animal food or feed) to which THC or CBD has been added. FDA is not aware of any evidence that would call into question these conclusions. Interested parties may present the agency with any evidence that they think has bearing on this issue. Our continuing review of information that has been submitted thus far has not caused us to change our conclusions. When this statutory prohibition applies to a substance, it prohibits the introduction into interstate commerce of any food to which the substance has been added unless FDA, in the - agency's discretion, has issued a regulation approving the use of the substance in the food (section 301(11)(2) of the FD&C Act [21 U.S.C. § 331(11)(2)]). To date, no such regulation has been issued for any substance. Ingredients that are derived from parts of the cannabis plant that do not contain THC or CBD might fall outside the scope of 301(11), and therefore might be able to be added to food. For example, as discussed in Question #12, certain hemp seed ingredients can be legally marketed in human food. However,all food ingredients must comply with all applicable laws and regulations. For example,by statute, any substance intentionally added to food is a food additive,and therefore subject to premarket review and approval by FDA, unless the substance is generally recognized as safe (GRAS) by qualified experts under the conditions of its intended use,or the use of the substance is otherwise excepted from the definition of a food additive (sections 20r(s) and 409 of the FD&C Act [21 U.S.C. H 321(s)and 3481).Aside from the three hemp seed ingredients mentioned in Question #12, no other cannabis or cannabis-derived ingredients have been the subject of a food additive petition, an evaluated GRAS notification,or have otherwise been approved for use in food by FDA. Food companies that wish to use cannabis or cannabis-derived ingredients in their foods are subject to the relevant laws and regulations that govern all food products, including those that relate to the food additive and GRAS processes. 11. In making the two previous determinations about,THC,why did FDA conclude that THC is an active ingredient in a drug product that has been approved under section Sog of the FD&C Act? In making the two previous determinations about CBD, why did FDA determine that substantial clinical investigations have been authorized for and/or institutedi and that the existence of such investigations has been made public? A.THC(dronabinol)is the active ingredient in the approved drug products, Marinol capsules (and generics) and Syndros oral solution. CBD is the active ingredient in the approved drug product,Epidiolex. The existence of substantial clinical investigations regarding THC and CBD have been made public. For example, two such substantial clinical investigations include GW Pharmaceuticals' investigations regarding Sativex. (See Sativex Commences US Phase II/III Clinical Trial in Cancer Pain (https://www.gwpharm.com/about/news/sativexr-commences-us-phase-iiiii- clinical-trial-cancer-pain) 2'(http://www.fda.gov/about-fda/website-policies/website- disclaimer)) 12. Can hulled hemp seed, hemp seed protein powder, and hemp seed oil be used in human food? A. In December 2018, FDA completed its evaluation (/food/cfsan-constituent-updates/fda- responds-three-gras-notices-hemp-seed-derived-ingredients-use-human-food) of three generally recognized as safe (GRAS) notices for the following hemp seed-derived food ingredients: hulled hemp seed, hemp seed protein powder,and hemp seed oil. FDA had no questions regarding the company's conclusion that the use of such products as described in the notices is safe.Therefore,these products can be legally marketed in human foods for the uses described in the notices, provided they comply with all other requirements.These GRAS notices related only to the use of these ingredients in human food.To date, FDA has not received any GRAS notices for the use of hemp-derived ingredients in animal food(see Question #25). Hemp seeds are the seeds of the Cannabis saliva plant.The seeds of the plant do not naturally contain THC or CBD.The hemp seed-derived ingredients that are the subject of these GRAS notices contain only trace amounts of THC and CBD,which the seeds may pick up during harvesting and processing when they are in contact with other parts of the plant. Consumption of these hemp seed-derived ingredients is not capable of making consumers "high." The GRAS conclusions can apply to ingredients for human food marketed by other companies, if they are manufactured in a way that is consistent with the notices and they meet the listed specifications. Some of the intended uses for these ingredients include adding them as source of protein,carbohydrates,oil,and other nutrients to beverages(juices,smoothies, protein drinks, plant-based alternatives to dairy products),soups, dips, spreads, sauces,dressings, plant-based alternatives to meat products, desserts,baked goods, cereals,snacks and nutrition bars. Products that contain any of these hemp seed-derived ingredients must declare them by name on the ingredient list. These GRAS conclusions do.not affect the FDA's position on the addition of CBD and THC to food. 13.What is FDA's position on cannabis and cannabis-derived ingredients in cosmetics? A.A cosmetic is defined in 2os(i) as "(1) articles intended to be rubbed,poured,sprinkled,or sprayed on,introduced into,or otherwise applied to the human body or any part thereof for cleansing,beautifying,promoting attractiveness,or altering the appearance, and (2)articles intended for use as a component of any such articles; except that such term shall not include soap." Under the FD&C Act, cosmetic products and ingredients are not subject to premarket approval by FDA, except for most color additives. Certain cosmetic ingredients are prohibited or restricted by regulation,but currently that is not the case for any cannabis or cannabis-derived ingredients. Ingredients not specifically addressed by regulation must nonetheless comply with all applicable requirements,and no ingredient—including a cannabis or cannabis-derived ingredient —can be used in a cosmetic if it causes the product to be adulterated or misbranded in any way.A cosmetic generally is adulterated if it bears or contains any poisonous or deleterious substance which may render it injurious to users under the conditions of use prescribed in the labeling,or under such conditions of use as are customary or usual (section 6o1(a) of the FD&C Act [21 U.S.C. § 361(a)]). If a product is intended to affect the structure or function of the body, or to diagnose,cure, mitigate, treat or prevent disease, it is a drug,or possibly both a cosmetic and a drug, even if it affects the appearance. (See Question #3 for more information about drugs.) FDA can take action if it has information that an ingredient or cosmetic product is unsafe to consumers. Consumers can report adverse events associated with cosmetic products via the FDA's MedWatch reporting system,either online or by phone at 1-86o-FDA-1o88, or by contacting your nearest FDA district office consumer complaint coordinator. For more information, please see the FDA's webpage on how to report a cosmetic-related complaint (/cosmetics/cosmetics-compliance-enforcement/how-report-cosmetic-rela ted-complaint). 14• Will FDA take action against cannabis or cannabis-related products that are in violation of the FD&C Act? A.The FDA has sent warning letters(/news-events/public-health-focus/warning-letters-and- test-results-cannabidiol-related-products) in the past to companies illegally selling CBD products that claimed to prevent, diagnose,treat, or cure serious diseases, such as cancer.Some of these products were in further violation of the FD&C Act because they were marketed as - dietary supplements or because they involved the addition of CBD to food. When a product is in violation of the FD&C Act, FDA considers many factors in deciding whether or not to initiate an enforcement action.Those factors include,among other things, agency resources and the threat to the public health. FDA also may consult with its federal and state partners in making decisions about whether to initiate a federal enforcement action. t5. Can I import or export cannabis-containing or cannabis-derived products? A. General information about the import/export of drug products regulated by FDA (/drugs/guidance-compliance-regulatory-information/import-export-compliance-branch-iecb) can be found online here.The Drug Enforcement Administration (https://www.dea.gov/)(DEA) is the federal agency responsible for enforcing the controlled substance laws and regulations in the U.S.and, as such,should be consulted with respect to any regulations/requirements they may have regarding the import or export of products containing cannabis. Please see here for information about importing or exporting food ingredients(/food/guidance-regulation-food- and-dietary-supplements/food-imports-exports). Regarding imports, if it appears that an article is adulterated, misbranded, in violation of section 505 of the FD&C Act, or prohibited from introduction or delivery for introduction into interstate commerce under section 301(11)of the FD&C Act, such article will be refused admission (see section 8o1(a)(3)of the FD&C Act [21 U.S.C. § 381(a)(M). Research and Expanded Access 16.What is FDA's role when it comes to the investigation of cannabis and cannabis- derived products for medical use? A.To conduct clinical research that can lead to an approved new drug, including research using materials from plants such as cannabis,_researchers need to work with the FDA and submit an IND application to the Center for Drug Evaluation and Research(CDER).The IND application process gives researchers a path to follow that includes regular interactions with the FDA to support efficient drug development while protecting the patients who are enrolled in the trials. For research for use as an animal drug product, researchers would establish an investigational new animal drug HNAD)file with the Center for Veterinary Medicine to conduct their research, rather than an IND with CDER. As discussed above(see Question #2), the 2018 Farm Bill removed hemp from the CSA.This change may streamline the process for researchers to study cannabis and its derivatives, including CBD, that fall under the definition of hemp,which could speed the development of new drugs. Conducting clinical research using cannabis-related substances that are scheduled by the DEA often involves interactions with several federal agencies.This includes: a registration administered by the DEA; obtaining the cannabis for research from NIDA,within the National Institutes of Health, or another DEA-registered source; and review by the FDA of IND of INAD application and research protocol.Additionally: • For a Schedule I controlled substance under the CSA, DEA provides researchers with investigator and protocol registrations and has Schedule I-level security requirements at the site cannabis will be studied. • NIDA provides research-grade cannabis for scientific study.The agency is responsible for overseeing the cultivation of cannabis for medical research and has contracted with the University of Mississippi to grow cannabis for research at a secure facility. Cannabis of varying potencies and compositions is available. DEA also may allow additional growers (https://www.federalregister.gov/documents/2016/08/12/2016-17955/applications-to- become-registered-under-the-controlled-substances-act-to-manufacture-marijuana-to) to register with the DEA to produce and distribute cannabis for research purposes. • Researchers work with the FDA and submit an IND application to the appropriate division in the Office of New Drugs in CDER depending on the therapeutic indication. Based on the results obtained in studies conducted at the IND stage, sponsors may submit a marketing application for formal approval of the drug. 17. Does the FDA object to the clinical investigation of cannabis for medical use? A. No.The FDA believes that scientifically valid research conducted under an IND application is the best way to determine what patients could benefit from the use of drugs derived from cannabis. The FDA supports the conduct of that research by: 1. Providing information on the process needed to conduct clinical research using cannabis. 2. Providing information on the specific requirements needed to develop a drug that is derived from a plant such as cannabis. In December 2o16, the FDA updated its Guidance for Industry: Botanical Drug Development(/regulatory-information/search-fda-guidance- documents/botanical-drug-development-guidance-industry),which provides sponsors with guidance on submitting IND applications for botanical drug products. 3. Providing specific support for investigators interested in conducting clinical research using cannabis and its constituents as a part of the IND process through meetings and regular interactions throughout the drug development process. 4. Providing general support to investigators to help them understand and follow the procedures to conduct clinical research through the FDA Center for Drug Evaluation and Research's Small Business and Industry Assistance group (/drugs/development-approval- process-drugs/oder-small-business-industry-assistance-sbia). 18. How can patients gain access to cannabis or cannabis-derived products for medical use through expanded access? A. Expanded access (/news-events/public-health-focus/expanded-access) is a potential pathway for a patient with a serious or life-threatening disease or condition to try an investigational medical product(drug,biologic, or medical device)for treatment outside of clinical trials when there are no comparable or satisfactory therapies available. Manufacturers may be able to make investigational drugs available to individual patients in certain circumstances through expanded access, as described in the FD&C Act and implementing regulations. 1g. Can patients gain access to cannabis or cannabis-derived products for medical use through Right to Try? A. Information for patients on Right to Try (/patients/learn-about-expanded-access-and-other- treatment-options/right-try) (RTl7)is available on our website. RTT is designed to facilitate access to certain investigational drugs through direct interactions between patients, their physicians and drug sponsors— FDA is not involved in these decisions. Sponsors developing drugs for life-threatening conditions are responsible for determining whether to make their products available to patients who qualify for access under RTT. If you are interested in RTT, you should discuss this pathway with your licensed physician. Companies who develop drugs and biologics,also known as sponsors,can provide information about whether their drug/biologic is considered an eligible investigational drug under RTT and if they are able to provide the drug/biologic under the RTT Act. Children and Pregnant/Lactating Women 2o. Does the FDA have concerns about administering a cannabis product to children? A. We understand that parents are trying to find treatments for their children's medical conditions. However, the use of untested drugs can have unpredictable and unintended consequences. Caregivers and patients can be confident that FDA-approved drugs have been carefully evaluated for safety, efficacy,and quality, and are monitored by the FDA once they are on the market.The FDA continues to support sound, scientifically-based research into the medicinal uses of drug products containing cannabis or cannabis-derived compounds, and will continue to work with companies interested in bringing safe,effective, and quality products to market.With the exception of Epidiolex, Marinol, and Syndros, no product containing cannabis or cannabis-derived compounds(either plant-based or synthetic) has been approved as safe and effective for use in any patient population, whether pediatric or adult. - 2t. Does the FDA have concerns about administering a cannabis product to pregnant and lactating women? A.The FDA is aware that there are potential adverse health effects with use of cannabis products containing THC in pregnant or lactating women. Published scientific literature reports potential adverse effects of cannabis use in pregnant women,including fetal growth restriction,low birth weight,preterm birth,small-for-gestational age, neonatal intensive care unit(NICU)admission, and stillbirth. [t, 2, 31 Based on published animal research, there are also concerns thatuse of cannabis during pregnancy may negatively impact fetal brain development. [4,5, 6 ]The American College of Obstetricians and Gynecologists(ACOG) recommends that women who are pregnant or contemplating pregnancy should be encouraged to discontinue cannabis use.In addition,ACOG notes that there are insufficient data to evaluate the effects of cannabis use on breastfed infants; therefore, cannabis use is discouraged when breastfeeding. [71 Pregnant and lactating women should talk with a health care provider about the potential adverse health effects of cannabis use. 22.What does the FDA think about making CBD available to children with epilepsy? A.The FDA has approved Epidiolex,which contains a purified form of the drug substance CBD, for the treatment of seizures associated with Lennox-Gastaut syndrome or Dravet syndrome in patients 2 years of age and older.That means the FDA has concludedthat this particular drug product is safe and effective for its intended use. Controlled clinical trials testing the safety and efficacy of a drug, along with careful review through the FDA's drug approval process, is the most appropriate way to bring cannabis-derived treatments to patients. Because of the adequate and well-controlled clinical studies that supported this approval, and the assurance of manufacturing quality standards, prescribers can have confidence in the drug's uniform strength and consistent delivery that support appropriate dosing needed for treating patients with these complex and serious epilepsy syndromes. 23• What should I do if my child eats something containing cannabis? A.With the exception of products such as the hemp seed ingredients discussed in Question #t2, which have been evaluated for safety,it is important to protect children from accidental ingestion of cannabis and cannabis-containing products. FDA recommends that these products are kept out of reach of children to reduce the risk of accidental ingestion. If the parent or caregiver has a reasonable suspicion that the child accidentally ingested products containing cannabis, the child should be taken to a physician or emergency department, especially if the child acts in an unusual way or is/feels sick. Pets and other Animals 24. I've seen cannabis products being marketed for pets.Are they safe? A. FDA is aware of some cannabis products being marketed as animal health products.We want to stress that FDA has-not approved cannabis for any use in animals, and the agency cannot ensure the safety or effectiveness of these products. For these reasons, FDA cautions pet-owners against the use of such products and recommends that you talk with your veterinarian about appropriate treatment options for your pet. Signs that your pet may be suffering adverse effects from ingesting cannabis may include lethargy,depression, heavy drooling,vomiting, agitation,tremors, and convulsions. If you have concerns that your pet is suffering adverse effects from ingesting cannabis or any substance containing cannabis,consult your veterinarian, local animal emergency hospital or an animal poison control center immediately. While the agency is aware of reports of pets consuming various forms of cannabis, to date, FDA has not directly received any reports of adverse events associated with animals given cannabis products. However,adverse events from accidental ingestion are well-documented in scientific literature. If you feel your animal has suffered from ingesting cannabis, we encourage you to report the adverse event to the.FDA. Please visit Reporting Information about Animal Drugs and Devices(/animal-veterinary/report-problem/how-report-animal-drug-side-effects-and- product-problems) to learn more about how to report an adverse event related to an animal drug - or for how to report an adverse event or problem with a pet food. 25. Can hemp be added to animal food? A.All ingredients in animal food must be the subject of an approved food additive petition or generally recognized as safe (GRAS)for their intended use in the intended species. If an animal food contains an ingredient that is not the subject of an approved food additive petition or GRAS for its intended use in the intended species, that animal food would be adulterated under section 402(a)(2)(C)(i) of the FD&C Act(21 U.S.C. § 342(a)(2)(C)(i)]. In coordination with state feed control officials, CVM also recognizes ingredients listed in the Official Publication(OP) of the Association of American Feed Control Officials(AAFCO) as being acceptable for use in animal food. At this time,there are no approved food additive petitions or ingredient definitions listed in the AAFCO OP for any substances derived from hemp, and we are unaware of any GRAS conclusions regarding the use of any substances derived from hemp in animal food. Learn more about animal food ingredient submissions(/animal-veterinary/safety-health/safe-feed)here. With respect to products labeled to contain"hemp"that may also contain THC or CBD,as mentioned above it is a prohibited act under section 301(11) of the FD&C Act to introduce or deliver for introduction into interstate commerce any animal food to which THC or CBD has been added. 26. Can approved human drugs containing CBD or synthetic THC be used extralabel in animals? A.The Animal Medicinal Drug Use Clarification Act of 1994(AMDUCA), permits veterinarians to prescribe extralabel uses of approved human and animal drugs for animals under certain conditions. Extralabel use must comply with all the provisions of AMDUCA and its implementing regulation at 21 CFR§ 530.Among other limitations, these provisions allow extralabel use of a drug only on the lawful order of a licensed veterinarian in the context of a valid veterinarian-client-patient relationship and only in circumstances when the health of an animal is threatened or suffering, or death may result from failure to treat. In addition, under 21 CFR 530.20, extralabel use of an approved human drug in a food- producing animal is not permitted if an animal drug approved for use in food-producing animals can be used in an extralabel manner for the use. In addition,under 21 CFR 53o.20(b)(2), if scientific information on the human food safety aspect of the use of the approved human drug in food-producing animals is not available,the veterinarian must take appropriate measures to ensure that the animal and its food products will not enter the human food supply. For more information on extralabel use of FDA approved drugs in animals, see Extralabel Use of FDA Approved Drugs In Animals(/animal-veterinary/acts-rules-regulations/animal-medicinal- drug-use-clarification-act-1994-amduca). [1] Gray, et al. Identifying Prenatal Cannabis Exposure and Effects of Concurrent Tobacco Exposure on Neonatal Growth. Clinical Chemistry. 2010; 56(9): 1442-1450. [2] Gunn, et al. Prenatal Exposure to cannabis and maternal and child health outcomes: a systematic review and meta-analysis. BMJ Open. 2016; 6:eoo9986. [3] Hayatbakhsh,et al. Birth Outcomes associated with cannabis use before and during pregnancy. Pediatric Research. 2012; 71(2): 215-219- 141 Silva,et al. Prenatal tetrahydrocannabinol (THC) alters cognitive function and amphetamine response from weaning to adulthood in the rat. Neurotoxicol and Teratol 2012; 34(1): 63-71. [5]Trezza, et al. Effects of perinatal exposure to delta-9-tetrahydrocannabinol on the emotional reactivity of the offspring: a longitudinal behavioral study in Wistar rats. Psychopharmacology (Berl) 2008; 198(4):529-537. - [61 Campolongo,et al. Perinatal exposure to delta-9-tetrahydrocannabinol causes enduring cognitive deficits associated with alteration of cortical gene expression and neurotransmission in rats.Addict Biel 2007; 12(3-4) 485-495• [7)ACOG Committee Opinion: Marijuana Use During Pregnancy and Lactation (https://www.acog.org/Clinical-Guidance-and-Publications/Committee-Opinions/Committee- on-Obstetric-Practice/Marijuana-Use-During-Pregnancy-and-Lactation) Gr (http://www.fda.gov/about-fda/website-policies/website-disclaimer) Related information • What You Need to Know(And What We're Working to Find Out)About Products Containing Cannabis or Cannabis-derived Compounds, Including CBD (/consumers/consumer-updates/what-you-need-know-and-what-were-working-find-out- about-products-containing-cannabis-or-cannabis) • FDA is Committed to Sound, Science-based Policy on CBD (/news-events/fda-voices- perspectives-fda-leadership-and-experts/fda-committed-sound-science-based-policy-cbd) EXHIBIT E MASSACHUSETTS DEPAR TMENT OF PUBLIC HEALTH CBD (Cannabidiol) in Ffod Manufactured or or in Massachusetts If I have a Massachusetts license or permit it legal to add hulled hemp seeds, hemp seed under 105 CMR 500 (Regulations for Good protein, and/or hemp seed oil to food I Manufacturing Practices for Food)to manufacture and/or sell at retail? manufacture food, is it legal to add cannabidiol (CBD) to that food? Yes. The _FDA has completed its evaluation of generally recognized as safe (GRAS) notices for No. The Massachusetts Department of Public hulled hemp seeds, hemp seed protein and Health (DPH) regulates food manufacturing inthe hemp seed oil. These products can be legally Commonwealth (see 105 CMR 500). These added to human food, provided they comply with regulations require that all food must be from all other requirements. approved sources that comply with federal, state, and local law and must not contain any prohibited I have a license issued by the Massachusetts ingredients. The FDA has concluded that federal Department of Agricultural Resources(MDAR) law prohibits the addition of CBD to food under its Interim Commercial Industrial Hemp products because CBD is an active ingredient in Program Policy. Is it legal for me to add CBD FDA-approved drugs. Since CBD is not an to manufactured or retail food? approved ingredient under federal law, it may not No. The MDAR Hemp Prooram does not provide be added to manufactured foods. an exception to the federal prohibition against If I have a Massachusetts permit under 105 adding CBD into food products. CMR 590 (Minimum Sanitation Standards for May I market my hemp products (e.g., hulled Food Establishments)as a food establishment, May I market eds, hemp seed protein and hemp eta ie?gal to add CBD to food I distribute at hemp seed oil) by making therapeutic claims without FDA approval? - No. The DPH sets minimum sanitary standards No. The FDA has reiterated that cannabis products - for food establishments in the Commonwealth. (hemp-derived or otherwise)that are marketed with These regulations are enforced by local boards of health in partnership with the state (see 105 claims of therapeutic benefit claim must be approved the FDAor any other disease CMR 590). The regulations incorporate the . FDA's Food Code, which requires that food be c products containing hemp cosmetic May I sell ti obtained from sources that comply with federal, or CBD ll retail Massachusetts? state, and local laws and must not contain any prohibited ingredients. As noted above, the FDA DPH does not regulate cosmetics. The FDA has has concluded that federal law prohibits the issued Frequently Asked Questions that address addition of CBD to food products because CBD is cannabis and cannabis-derived ingredients in an active ingredient in FDA-approved drugs. cosmetics. Since CBD is not an approved ingredient under federal law, it may not be added to foods - What can I do if I had an adverse reaction to a distributed at retail. cosmetic product labeled as containing CBD? If I am licensed or permitted as a food You can report a cosmetic related complaint manufacturer or retailer in Massachusetts, is directly to the FDA. ' Massachusetts Department of Public Health Bureau of Environmental Health y7r 250 Washington Street, 7th Floor, Boston, MA 02108 aV,�gy LVE Phone: 617-624-5757 1 Fax:617-624-5777 1 TTY: 617-624-5286 .�, June 2019 EXHIBIT F asao34q TOWN OF NORTH READING BOARD OF HEALTH 11iROBERT F.BRACEY Bla[ROn ov[Deaf Henvx RGOINO Tmm HALL-ri N.— "Narl•N.—RUOINC,MA.OIW Ari TI r res.9laJ5A5N3 PublicHt'alfh aAcer Y Nma.Acov June 24,2019 RE: MANDATORY REMOVAL OF ALL CBD FOOD PRODUCTS Dear Business Owner/Operator: Be advised that this letter contains important information. This letter and auschment explains the Massachuni Department of Public Health Food Protection(DPH)and the US Department of Health nod Human Services Food and Drug Administration(FDA)policy regarding hemp-derived cannabidiol (CBD)in food manufactured or sold in Massachusetts,which will be enforced within the Town of North Reading As of June 17,2019,all businesses,including licensed food establishments,operating within the Town of North Reading are prohibited from producing,manufacturing advertising,offering or selling any food or other consumable products that contain CBD. Such products have been expressly prohibited the FDA a well as the DPH, ee htlos//wwwrase gov/info-datailslcbd-iii-food-manufactured - Id= - massachuseus The Health Department will be strictly enforcing the DPH and FDA policy guidance in this arca starting July 1,2019.Any licensed food establishments or other business operating within the Town found to be in noncompliance with the DPH and FDA policies shall be subject to penalties and fines in accordance with the Health Department Rules and Regulations Chapter 1"Administrative Procedures": SECTION 19 PENALTIES 1.19.001 Any persons,firms,or corporations violating or failing to comply with any provision at my rules and regulations of the Board of Health,shall be warned or fined in accordance with the noncriminal disposition process adopted by the North Reading Board ofSelectman on October 6,1980 Town Meeting-Article 6 1st offense—written warning or one hundred(100)dollar fine 2nd of7evse—two hundred(200)dollar fine 3rd offense—three hundred(300)dollar fine 4th and subsequent offenses—three hundred(300)dollar fine daily and/or other enforcement action the Board of Health deems necessary including but not limited to suspension and or revocation of permit to operate in the Town of North Reading If you have any questions,please feel fiee to correct the Massachusetts Department of Public Health Food Protection Program ar 617-983-6712 or FI'POPH(ustale ma us or the North Reading Healtb Deportment at bbracevnm nouhread'nnma Roy or 978-357-5242 Sincerely, Robert F srmcaH Robert F.Bmccy Health Director Cc:Board of Health Director of Public Safety THE YALE a DANIEL E. NO Fudging the Nudge: Information Disclosure and Restaurant Grading ABSTRACT. One of rhe most promising regularory cuaenm<oamisrs oftargeted'disclosure: mandating simplified information disclosure at rhe time of decisiomnaking to"nudge" parties along.Its poster child is munionm saturation grading.In principle,a simple posted letter grade ('A,"B;or'C')empowers consumers and properly incemivi=restannteurs toreduce risks for foodborne illness.Yet empirical evidence of the efficacy of resunram grading is space This Article fills the void by studying over 700,000 health inspections of restaurants across ren poisdictioam,forming on San Diego mid New York.Despite grading's great promise,we show that die regulatory design,implementation,and paactim suffer from serious flaws:jurisdictions fudge more than nudge.In San Diego,grade inflatio a reigns.Nearly all restaurants receave'A's. In New York, inspections exhibit little substantive consistency. A good some does nor me mingfilly predict cleanliness down the road.Unsurprisingly,New York's implementation of letter grading in min las not distenaably reduad manifestau ar s of foodborne illness.Perhaps worse, the system perversely shifts inspection resources awav from higher healdn haaerds to The.resolve grade disputes.Theresults have considerable impllmtions,not only for food safety,but also for the insdadonal design ofinf rmauon disclosure. AUTHOR, Professor of Law, Smrdord Law School; E-mail: dho@lma.stanaatd EI an Stvaa hap://dho.stmffbrd.eda.Thanks m Patrick Leahy,Modula Raman,JsSeek, for excellent research assistance, and Brace Ackerman,Jemufer H. Arlen, Ian Ayres,Vicki L Been,Conti Ben-Shahar,Rolaea Bookman,Ryan Bubb, Martimm-Flotentino CO61lar,Micharl C. Dorf, Ted Eisenberg, Robert C. Ellickson, David Freeman Engstrom, Michel Frakes, Stephen Galoob,Jonah B.Gelbada,Heather Gerken,JacoClnisune oilAbbe K Gluck,Valee Hans, Do=A.Hathaway,Michad Heise,Robert A.Hillman, J Y Pay es Kaplow,Mark Kelman,Jon Kuck,Mitchell Lassa,Daryl Levinson,Daniel Mmkovits,Morencia Hammet-Wurgler, Bernie Meyler, Alison D. Morana, Nicholas Perin., Robert C. Post,Jeff Rach inski,Richard L.Revesr,Roberta Rommuo,Susan Rose-Ackerman,Jed Rubenfeld,Steven Shavell,Bill Simon,Laura Trice,Tom Tyler,David N.Weil,Jolts Fabian Witt,and participants at die Administrative Law Roundtable at Columbia Law.School, the law and comments workshops at ETH Zurich and Harvard Law School, and die faculty workshops at Cornell, Stamford, and Yale Law Schools for helpful comments and conversatiom,Q.Ethan McCallum for sharing data, Tolga Ergmay and Darin Phelps for computational assistance, and Mike Vmi&rHeijden and Sarah Kraus for help with FDA sonrces. The empirical work was collaborative,so dus Article refers to the research team in die pled. 574 OPP PP n ARTICLE CONTENTS INTRODUCTION S77 1. THE LANDSCAPE OF GRADE REFORM 588 A Historical Antecedents 588 B.Public Health Doubts 591 C.Los Angeles Faith 595 II. EMPIRICALLY ASSESSING GRADING 599 A.The Confounding Nature Of Grade Reform S99 B.jurisdictional Variation 6oL C.Our Approach 6o6 III. FUDGING BY INFLATION: SAN DIEGO 607 A.Regulatory Background 607 B.Inspections Data 610 C.Bmpirical Results 631 L.Uniform Gmde Inflarion 611 a.Scoring Consistency 613 IV. FUDGING BY NOISE: NEW YORK 616 A Regulatory Background 617 L.Inspections 2005 to 2010 617 2.The 2009 Comptroller Audit 622 3.Letter Grading 622 4.Internal Assessment 625 B.Inspections Data 626 C.Empirin!Results 629 I.Grade Variation 629 2.Grading Changes Scoring 632 3.Scoring Inconsistency 635 V. EXPLAINING THE SCORING DIVERGENCE 638 VI. INTENDED AND UNINTENDED EFFECTS 643 575 THE YALE LAW JOURNAL Jan:574 2012 A Health Outcomes 643 B.perverse RCSOnrm Allocation 647 VII. IMPLICATIONS 650 A.Design Matters 65o B.Retargeting Transparency 654 CONCLUSION 656 APPENDIX b58 A DOHMH Data IOwgrity 658 1. December Version 658 z. December Version and DOHMH Site 659 3. Comparison of Five Versions 66o B.Classification Algorithm 661 C.Robustness 664 r.Types of V solations 664 a.Consistency in Other Types oflnspections 665 3.Administrative Hearings 667 4.Random Sample from die DOHMH Website 670 D.Corroborating Evidence from Eight Other Jurisdictions 67- E.Sources for Tables 1&a 677 576 FUDGING THE NUDGE INTRODUCTION When does disclosure work? Mandated disclosure to Solve informational failures—and to empower parties to make informed decisions—has long been recognized as a theoretical matter.' Examples of mandated disclosure abound across regulatory areas as diverse as securities regulation,' campaign finance,` product safety,° energy regulation,` employment law,` environmental law,z and health law.' Yet despite the fact that disclosure is a mainstay of the regulatory toolkit, a fierce debate persists about the conditions under which disclosure works.9 v See,e.g.,STEPHEN BREYER,REGuuiwN AND ITS REFORM z6-28, 161-64 (1982);JOSEPH E. STIGLITL,ECONOMICS OFTHEPUBLICSECTOR 83-84(3d ed.2000). a See Securities Act of x933,Pub.L NO.n-22,48 Star.M(codified as amended at 15 U.S.C. 55 na-n.(2006)). 3. Sar Federal Election Campaign Act of 1971, Pub. L. No. ga-x25, 86 Son 3 (codified as amended at 2 U.S.C.55 431-56(2oo6)). 4. See Consumer Product Safety Improvement Act of zoo8,Pub.L No.110-314,m Star.3o16 (codified in scattered sections Of 15 U.S.0 (20o6)); su also Publicly Anulable Consumer ProdnR S#ny I f ration Database, U.S. CONSUMER PRODUCT SAFETY COMMISSION, hap://wwwsafetprodncts.gov(last visited Jan 30, 2-La) (providing a publicly ttarduble database where submlaers ran report a harm or risk of Kann related to Else use of a consumer product or substance). s. See Guide Concerning Fuel Economy Advertising for New Anmmobiles, 16 C.F.R. 55 2591-259.2(aou)(mandating disclosure of estimated dry and highway miles per gallon for aummobaes); id 5 305.11 (mandating disclosure of energy consumption and waver usage for appliances); see also Energy Sar, U.S. ENVTL PRDTECFFON AGENCY, http://www.mergysw.gov(last visited Jan.30,zolz)(describing a joint program with the Environmental protection Agency and Deparmmnt of Energy concerning labeling and environmental standards for products aid buildings). 6. See OSTIA Hazard COmmurdcarion Standard, 29 C.F.A. 5 1910.1200 (2011) (mandating disclosure ofharardons chemicals ro employees and employees). z. See Emergency Plaurdng mid Community Right-To-Krim Act of 1986,41 U.S.C.55 Hoot- Hop (z o6) (requiring disclosure of chemical hazards in communities); Standard for Demolition and RenowdoR 40 C.F.R. 5 61.145(zou) (mandating disclosure of asbestos- releasing demolition and renovation activity);id 5156.10(mandating disclosure ofpesticide ingredients); Mandatory Reporting of Greenhouse Gases, Tq Fed. Reg. 56,26o (proposed Oa. 30, 2oo9) (to be caddied in scattered pares Of 40 C.F.R.) (requiring reporting of greeudi0use gas emissima). e. See a C.F.R. 5 101.9 (2011) (regarding the disclosure of nutritional information for food products);4L 5 141.r(concerning the display of healdl warnings on padages of cigarettes and cigarette advertisements). 9. See JAMES T.H VauTON,REGULATION THROUGH REVEtaTION:THE ORIGIN,POLITICS,AND LNPACTS OF THE TOXICS RELEASE INVENTORY PROGRAFI(2005);W.KIP VLSCUsi&WESLEY A. MAGAT, LEPRNING ABOUT RISK: CONSUMER AND WORxER RESPONSES TO HA]ARO 5n THE YALE LAW JOURNAL Ixx:574 2012 Over the past few years, one of the most important regulatory developments has been the emerging focus on "targeted transparency." The chief insight, based on behavioral research, is that the public faces significant cognitive limitations in processing information." More information is not always better." Instead, effective forms of regulatory disclosure are "targeted": simplified disclosures embedded at the point of decisionmaking to "nudge" parties along." In the terms of Cass Sunstein and Richard Thaler, INFORMATION (1987); Own Bar-Gill & Franca Ferrari, Informing Qrs mom Mout Themnlves,3 ERASMUS L REV. 93 (x010); Marianne Bertrand&Adair Morse, Information Disdomm, Cognitive Biases,and Payday Borrowing,66 J.FIN. 1865(2ou);L<emore,Dafny& David Dranove,Do Report Cards Tell Comumers Anything They Don't Already Knmo?The Case ofMedinew HMOs,39 RAND J.EGON.790(mo8);David Demovc et al.,R More I fmnolfon Better?The Effla of"Report Cards'on Health Care Providers, m J.Pot.EMN. 555(2003); Allen Ferrell, Mandatory Disdmure and Seadc Returns: Evidence from the Over-the-Counter Market, 36 J. LEGAL STUD. n3 (xo07); Kduel J. Fishman & Kadalee. M Hagerty, Mandatory Versus Voluntary Disclosure in Markets with Informed and Uninfonnd Custameo,a9 J.L. Boom &ORD. 45 (x003); Roberr, A. Hillman, Online Boderpdum: Would Mandatory Webster Disclosure of E-Standard Tem,Backfire?, 104 MICH.L Rev. 837(2006); Shameek K..&Matic A.Cohen,Ii[notion m Reguh cion: The Effect of community Right To Know Laws on Toxic Emi,imu,32 J. ENvrL.EWN.&MGMT. 109 (1997);Anthony T.KcIance n, Muurke,Duelnum,Ialirmation,and the Law ofContram,7 J.LEGAL STUD.1(1978);Paul G. Malwltey,Marilee ry Duclomre u a Solution to Agency Holden,, 6x U. CHI. L.REv. 1047 (1995); FlomFd,Mm1a-Wurgler,Doo Contract Disclosure Mann?, 168 J.INSTITUTIONAL &THEOREncAI EGON. % (2012); Florencia Marotta-Wurgler, Will Btneasal Disclosure Help?Evaluating the Recommendation,ofthe AL1}"Priruiphs ofthe Law ofSofmwre Com"Aft, 78 U.CHL L REV.165(m11);Alan D.Ma Bios,The Impact of Mandatory Disclosure Lowy on Product Choim:An Analysis of the Sabel Dressing Market, 43 J_L.&BOON. 651 (woo); A Mitchell Polinsky&Steven Shaodl,Mandatory Fonts Vobnmry Duclmure of Prvdua Risks, x6 J.L.Boom&ORG.1(2010);William M.Sage,Begdtuing Through Infrmation:Disclosure Laws and America[; Health Care, 99 COLUM. L. REV. 17o1 (1999); Dmuel J. Solove, The VA. of Xnat,trg Los:Justi ging Privacy Prornums Against Docbsare, 53 DUCE L.J. g67 (2003);Now,Disclosure as a Legidadve Devic,76 HARv.L.Mv.1x73(t963) .o. Sm, e.g., Jan. R. Beeman et al., Cognitive Com3dermans in Designing Eff afte Labels for Haen,og Auk InlAUnm-n,5 J.PDB.POI:v&MA uUmNG 1,x-1x(1986). U. Sm, eg.,Eugene G. Chewning,Jr. &Adrian M. Harrell, The Effect of/.motion Load on Decision Makes' Cue Unification Levels and Derision Quality in a Financial Dismar Donnan Task,15 Acer.ORD.&SOC'Y 527.539-40(1990);Kevin Lane Keller&Richard Sraelin,Effecs ofQmafty and(tummy oflnformaMn on Doman EfjctivencU,14 J.CONSUMER RES.x00,x;- 12(1987). n. The Seminal Synthesis it ARCHON FUNG,MART GR.IHAM At DAVID WEIL,Full.DISCLOSURE: THE PERIIs AND PROMISE OF TRANSPARENCY (2007). See alto ROBERT BALDWIN ET AL., UNDERSTANDING REGULATION: THEORY, STRATEGY, AND PRACTICE 339 (2d ed. 2012) ("[OJne ofthe key debates in recent years las been how'argeted transparency'tan be used ro inflownce (or'nudge') mosumer behaviour in order to rectify the limitations of more conventional mid poetical-buremcatic undersnmfings of mounubiliry."); RICHARD H. THALER&CASS R.SUNST£IN,NUDGE:IMFRONNG DE IS10Ns ABOUT HFALTH,WE4LTH,ANO 578 FUDGING THE NUDGE interventions should focus on structuring choices to nudge parties toward decisions that are less prone to heuristics and biases of decisionmaking.13 Age grading of.children's toys,° star ratings for SW rollover risk,' and smart energy meters 16 arguably embody such prescriptions. The Obarna Administration has embraced targeted transparency and HneewEss(2008)(proposing a framework for choice architecture no"nrrdge'parries toward more effeetiec decisimtmaking); Omri Ben-Shahan & Carl E. Schneider, The failure of M.u&td Denbsure, 59 V. P, L. REv. 647, 65x. 743 (zmQ (arguing dial "mandarcd disclosure generally ails to achieve it goals,"but chat"brief,simple,easy disdosures"and ratings provide a promising regulatory alternative); Colin Carteret et al., Regulation for Conservatives:Behavioral Bramanc,and the Case far "Asymmetric Peemudsm,'151 U.Pa L REV,rue,uv,1230-32(1003)(noting char"(tlhe goal of asymmetric paternalism is to help boundedly rational consumers make better decisions'and discussing disclosure regulations that further that goal);David M.Grether,Alan Schwar¢&Louis L.Wdde,The Irrekwry of Iforaadon Onaload: An Analysis of Search and Ducbsure, 59 & Gu. L Ram. 277, 301 (1986) (noting that mruin forms of samficiag "cart be ameliorated by disclosure requirement that reduce the costs to consumers of inspecting product attribures");Samuel InaeharolE, Diubmre, Agent, aId Camumm Proveaio., r67 J. INSTITUTIONAL. & TnnoRanaa EGON.56,63(tom )("Ofaiucal impotence is not only the consumer's ability to obtain relevant information through disclosure,but that the information be ofa sort that will prove usable within real-world time and motivation constraints.");Cluirtum Jogs,Cass IL Summin&Richard Thaler,A Behavioral Approach It Law and fr oromia,So STAN.L mEv. 147r, 1533-3n, (1998) (discussing how conventional prescriptions to 'provide more information" fall short when considering behavioral insigho); Christine jails &Cass R. Summit,Debhsig Through Law, 35 J LEGAE STUD. 199, 207-08 (2006) (ualyzing legal strategies for reducing the effects of bounded rationality and discussing in¢muodiate options to greater information disclosure and outright barn); Mario F. Teid & Brian Roe, Zhu Ereewmia aflnbeliag:An Overview of/nu s for Health and Environmental Duclosurs,27 AGIuc& REsouracE EcoN.REm.140,W-47(199g)(Providing a theoretical framework for rhe welfare analysis of itformaton larding that poses a tradeoff between information cost azul accuracy); David Weil et al., The Effwavmas of Regulatory Duclasure Policia, 25 J.Poi T ANN.YSIS& MGMT.155(2oo6)(reviewing p nviples ofef[ceme disclosure arross regulatory areas). u. See Tn k&SVNSTEIN,supra rare Iz,at 11-13(arguing,for crumple,that designers should give chooses mmindus and try to minimize costs for those who do not want on choose). is. See Age Determinant. Grud fmau Rdming Childrm i Aga a Toy Characteristic and Pry Behavior,CONNMER PRODUerS S"M COMMISSION(2002),hap://www.cpsc.gov/businf, /adg.pdf. is. See Tunsporation Recall Enhancement,Acme inability,and Documentation(TREAD)Are Pub.L No.Io6-414,114 Sat. 180o(z000) (codified as amended in scattered sections of 49 U.S.C.(z000)). ,s. See Rebeaa Smith, Smart Meter, Dumb Idea?, WALL ST. J., Apr. 27, 2009, hap://odine.cos}.mm/artidc/SB.z oS"4 6144 { 855S.hmd;Clive Thompson,Cbm Th.pson Thinks: Derkmy Orb Could JZfun, Eneir Hogs, WIRED MAG., July 1, x001 http://www.wired.roto/rechbiz/people/magazine/15-o8/st_thompson; News Release, Cal. Pub. Ural. Comm'., PUC Approves Smarunemrs for PG&E Customers (July 20, 2006), hay://dors.cpnc.a.gov/NnblishedDocs/Word_PDF/Nms_Rdease/58233.pdf. 579 THE YALE LAW JOURNAL 122:574 2012 behavioral insights in its regulatory approach,'most notably in the appointment of Cass Sunstein as Administrator of the Office of Information and Regulatory Affairs (aka. "nudger in chief"").Executive Order 13,563,which reaffirms cost- benefit analysis of proposed regulations and mandates retrospective review of existing regulations,champions"provision of information to the public in a form that is dear and intelligible."'9 In a series of memoranda to agency heads, Sunstein further refined the Administrations approach: "Agencies should consider how best to eliminate unnecessary complexity and to simplify people's choices."'' Information technology and intermediaries should serve that end, with agencies encouraged to release "complex information and data in standardized, machine readable formats (to] enable consumers to nuke informed decisions."" In zoH, the Administration convened a National Science and Technology Council Task Fora on"Stuart Disclosure.— Agencies, in turn, have developed a host of proposals in line with targeted transparency. The Securities and Exchange Commission promulgated a rule requiring standardized (machine-readable) risk-return summary disclosures for mutual funds." The Environmental Protection Agency and National Highway Traffic Safety Administration proposed motor-vehicle letter grading n. Sea Michael & Livermore &Richard L. Revea, Address, Bending Rationality Two Yam Ino,48 HOUS.L.REv. 1,19-23(aou)(discussing the Obama Admwstration's rcfirtemen, of cost-benefit analysis m incorporate insights from behavioral ecommies); Michael P. Vandenbergh,Amanda R.Carrico le Lisa Sehule,Bressman,P-rda an in the Behavioral Bra, 95 MINN.L.REV.573,717(2011)("Regulation las entered the behavioral era.");cf.Charles F. Sabel&William H. Simon,Minimalism and Erperimrnwlism in the Admin6bative Sink, too Geo.L.J.53(aou)(critiquing minimalist theories of regulatory policy). a. Jeff Sommer, Man Humare,Need a Nudge Toward Rationality,N.Y.Tunes, Feb.7, 2009, hap J/wwsv.nytimes.mm/aoo9/oa/oe/bnsincss/oenndgehuN. h9. 1 Order No.J3,563,76 Fed.Reg.3,8214 4(2011). Jr. Memorandum from Cass R Smutein,Adm'q Office of Info.Be Regulatory Affairs,for die Heads of&ret. Dep'ts &Agenees, Disclosure and Simplification as Regulatory Tools u Dune 18,2010). .,. Memorandum from Cans R Sunsteim,Adm'r,Office of Info.&Regulatory Affairs,for the Heads of Exec. Dep'ts & Agencies, hdivering Consumers Thmugh Smart Disclosure 2 (Sept.8,201 ). s:. Memorandum from Aneesh Chopra,U.S.ClvefTerh.Officer&Assoc.Dir.for Teelr.,Office of Sc. &Tech. Policy,for die Nat'l Sci. &Tech.Council Comm.on Tech.,Winning the Furore Through Open Imanarion—A Progress Report on Our Open Govvnmem Initiative 2 Dune 8,2011). J, Sea Interactive Data for Mutual Fund Risk/Renun Summary, m Fed.Reg. 7,748(Feb. 19, aoo9)(to be codified at p O.F.R.pts.230,a3a,239&2�). 580 FUDGING THE NUDGE for fuel economy and greenhouse gas emissions."i The Food and Drug Administration simplified sunscreen labels to minimize consumer confusion." The Department of Housing and Urban Development issued a grant to the Center for Neighborhood Technology to create a national Housing and Transportation Affordability Index.16 And the Consumer Financial Protection Bureau is currently experimenting with simplified mortgage disclosure fortes." The central ideas of targeted transparency continue to inspire scores of normative proposals.'a y. See Recisions and Additions en Moror Vehicle Fuel Economy Label, 7S Fed. Reg. 58,078, 58,082(Sept a3.2010) (proposing lever grading);76 Fed. Reg.39,178,39.488-89 Uuly 6, 2011)(ev be codified at 40 GF.R.pts.85.86&600,49 C.F.R.pt SS)(choosing alternative label); Fol Ecnnamy Label: Expert Panel "a, U.S. ENvrL. PROTEcnON AGENCY (Aug. arm), step..//www.epa.gov/hmleconomyAabcV4zorw9o8.pdf (nuking design oxommendatia.for letter-grade labels). aS. Sar Labeling acrd Effectiveuss Testing; Sunscreen Drug Produces for Over-the-Counter Human Use,76 Fed.Reg.3S.62o.35624(June 17,2011)(m be codified at 21 C.F.R.per art &3m). a6. Sec Press Release, U.S. Dep'c of Hous. &Urban Dc, HUD Lanncbes Development of a National Housing and Transportation Affordability Index: New Tool Will Provide Homebuyers with a More Accurate Estimate of Living Costs (Aug. 30, 20u), littp://porLo.hud.gov/hudpormVHUD?sm=/press/press_releascs media_advisories/amy HUDNo.r1-reo. 2/. See Knew Before You Owe, CONSUMER FIN. PROTECTION BUREW, http;// www ,ourro m aImance.gov/lmowbeforeyouowe(last visited Jan.30,2012). a9. Set,eg.,LIN AYRFS&JENNIFER GERPRDA BROWN,STRNGHTMRWMU:HOW TO MOBWZE HETEROSEXUAL SUPPORT FOR GAY IUGHTS 79-94 (an-5) (prop0.5mg, a"Fav Employmrnt Mark' to certify Thu an employer abides by a set of cmploymem policies); HEATHER K. GERKEN,THE DEMOcsa Ca INDEX;WHY OUR ELECTION SYSTEM IS FMUNOAND HOW TO FIX IT IM (a-09) (Proposing a "Democracy Index" dun would create "better information shortcuts for Yorers,politymakers,and election administramrs"to improve die U.S.eketion sport); LAWRENCE LessIG,R,,UoUC,UJST: How MONEY CORRUPTS CONGRESS—AND A PLsm To STOP IT a51-63 BE au(2011) (discussing how campaign finions,by failing tenets of targeted tramp rcency,will not for "dependence corruption" in U.S. politics); Omd Ben- Shahaz,The Myth of the'Oppunamry To Read'in Comae law,S EUR.Pev.Cow.L.1,a1-a6 (zoos) (proposing aa simple,intuitive formai for rating and labeling ofconvuts);Nora Freeman Engstrom, S,,.UgM and Sadoarm Milh, 86 N.Y.U. L. REv. 805. 865-71 (amQ (proposing a disclosure regime for settlement mills that operate in die shadow of ton law and discussing methods on facilitate client use);Cynthia Esdund,jest the Facts:The Cane for Workplan Traxnpa en}, 63 STAN. L REv. 351. 37699 (am ) (proposing a mandatory disclosure regime for the workplace and discussing die "ingenious"c0naption of targeted trampamo,y);jeff Leslie&Cass R Summit.Animal Rights Without Cmmroversy,TO low& CONTEMP. PROBS. 117, 13o-36 (2007) (discussing proposals fol simplified animal welfare products labels); Kristin Madison, Hie tam and Polity of Hmhh Care Qmhty Rryor[ing,31 CAMPBELL L. Rev. 215, 21-52 (2009) (discussing efforts to improve Health care quality reporting); Lloyd Hiroshi Mayer, Duciesuro Abes,Disn4viao,,WD.L.REv. 251, 17-6-, 280-84(tom) (discussing flawed aucmpts of campaign finmce disclosures and proposing 581 THE YALE LAW JOURNAL 13x:5)4 ants Targeted transparency's poster child is restaurant sanitation grading. The central idea is to summarize sanitation inspections with letter grades ('A,' B,' or'C) and post these in entryways of restaurants to succinctly and intuitively inform consumers.In theory, the disclosure helps consumers select restaurants based on health risk, which in turn incentivizes restaurants to clean up. In the seminal synthetic work on targeted transparency,Archon Fung,Mary Graham, and David Weil systematically review disclosure policies and associated empirical research across regulatory areas, finding restaurant hygiene disclosure to be one of two"highly effective" regimes because of its simplicity and comprehensibility.�9 Indeed, Fung, Graham, and Weil use restaurant grading as the motivating example of how to "embed" disclosures with individual decisiontraking in an informative and comprehensive fashion-l" Restaurant grading, according to them, exhibits congruence between policy and consumer goals m reduce food-poisoning risk, with only a moderate chance of misinterpretation." Similarly, Oran Ben-Shahar and Carl Schneider argue chat mandated disclosure has generally been a failure across policy areas, but they point to restaurant grading as a salutary exception and as the prototype for promising regulatory alternatives." Restaurant grading is widely considered a paragon of disclosure regulation.33 In a landmark study, Ginger Jin and Phillip Leslie reported that the adoption of grading in Los Angeles in 1997 caused a zo% reduction in ro change die level of disdosures); Gideon Pardiomovsky&Philip J.Weiser,B,w d Fair Use, 96 CORNELL L.REv. 91, 96, 116 (a01o) (proposing ea mandate[liar content owners provide dear aid explicit notification of user privileges); Richard B. Stewart, A Nav Generation ofEnvirownenrd Regulation?, 29 Cnv. U. L R£v. 11, IM-43 (1001) (disalssing ilderseational strategies for environmental regdadon);Andrew Buick&Andrew Cancer, Now, Supply, D Lind, and he Clanging Economics of Inrge Law Fimu, 60 STAN.L Rev. 2087, u18-26 (ions) (describing dao-driver efforts bued on rargeted transparency to provide rankings of law films based on hours,diversity,and araitiOR). 29. FUNG ETAL.,supra nate q at 82-83. so. Id at 5o-83. p. See i&a,y5. 3r. See Ben-Sliahnr&Schneider,supra now la,ap43-48. 33- See, e.g., ORG. MR ECON. Co-OPERATION&D£v., CONSUMER Popsy ToOLRrr 87 (aOIOJ; THALER&SUNSTEIN,supra now 12,at 19o;Paola J.Dalley,The Use and Mine ofAwlarum as a Regulamry System,34 FLA.ST.U.L REV.Joss,rug (a J07);Estlund,supra now,28,at 394-95; Issuharoff, supra now 12, a 6o; Frederick Schauee, Trawparenry in Tree Dlme,Lnom,ion U.Iu.L.REV.1339,1348;Uor Jamb Smalidevia,Rep,axion Nation:Law in an Era of U6iquieom Pusan( Ifonmrion, ins Nw. U. L. REv. 1663, p11 (1008); Find Ecmamy Label Expm Panel Report,supra note ay,au6. 583 FUDGING THE NUDGE hospitalizations for foodborne illness 14 Mayor Michael R. Bloomberg, who introduced restaurant grading to New York City in July nolo, called grades "wildly popular"" and concluded that as a result "the City made restaurants cleaner, safer and more transparent."36 Forbes magazine described New York's system as"The Most Effective Regulatory Disclosure Ever."37 Other jurisdictions, in turn,have jumped on the bandwagon.Over the past ten years, in addition to Los Angeles and New York, Georgia,}8 Hartford,1° Louisville,"' Mississippi°' Toronto," Albany County (NY),"' Cuyahoga m. Sto Ginger Zhe Jim&Phillip lrslie,The Efea oflrfornwtion on Product Quality:Evideru'ef m Restauwu Hygiene Grade Cards,its Q.J.ECOM 409(zo03). 3s. David Seiformh, Mika N Health Dept. at Food-Cart Feud, N.Y. Porn, Aug. z -it, ]Imp..//www.nypost.corn/p/news/local/mike_healrh_dept_in_food_carr_fnsd_yCF8S18a,4J QTFAuAMBnJ. a6. press Release, Of o, of dee Mayor, Mayor Bloomberg, Depnry Mayor Gibbs, Health Commissioner Farky Announce that Onc in Three Resunturn in New York City Aummatidly Saved Money by Earning and Keeping'A's—Over$3 Million in Fi ib st over laSix Months (Aug. h, 2011), htM://www.nycgov/luml/om/bmd/am r /pa e -hr.html. 3r. Kai Falkenberg, The Man Elective Regirktary Derrlosma Ever. So Easy Even Toddlers Understand It,Fonass,May 6,zou,http:/frsvv+v.fortes.mm/sites/kaifalkenberg/zou/oS/o6 /dee-.o ,ffemd -regulamrydisdosure-ever-so-eusy-even-mddlers-wrdeamnd-iS see aho Tom Ferrick,Jr., How Nov York Geo Food Impeaiom Right, M[Txoeous, Nov. 13,nom, Izup;//www.pNmetropolismm/zozo/ts/how-new-york-gets-impecdow-righcphp("If you want ea see restaurant wepection, done right, ravel to New York City"); Elisabeth Rosenthal,I Diselm,...Nothing,N.Y.Tures,Jan. 21,20rz,http://www.nyt m=.wm/zoz /oi/v/sunday-miew/hard-oaths-aboutdisdosum.html (m ing that restaurant grading is considered a hc1pfid form of disdosnre,while most disclosure policies may net work). 36. See Eliraberh Ur,Renauram Free New Rating Code,ATutrrA J.-CoNsr.,Dm s,2o07,At Jh. 3g. See Jeno Cazlesso, Hartford Retowevars To Get Health Grades, HARTFORD COUantd3', Dec. 31, zou, http://ar6cksmnrytt.mm/2011e2-3s/business4K hadord-restaurant-sconng -tzzs-zoo hu3_r_inspec6on-restnnnnts-food-smrage-and-preparaton. 4a. See Gideon Gil,New Rating System Set for fefmvo Resmmanu,COURIER-J.(Wuisvtle.Ky.), Dec.3,2ooz,at Bh. 4.. See Mississippi State Department ofHeakh Annourues New Restaurant hvpeawn Rating System, Gure COASTNEw5(Miss.),Sept ho,zooy,huP://www.gn'fcoastnem w. ,/gc=rcwve/zooy /grn msncwnesmurmchealdrradngs091007.hum. 42. See Yvonne Blackwood,E nth for Rea rmorm Inspection System, Toncamim SrnR,Jan. u, zooz,at Azo. q3. See Sovc Barnes, New Albany Restaurant-Impertwn Geodes: IlaadLm, Good, Fair, TufFs UNION: TAet6 HOPPING (Alba y, N.Y.) (Jan. u, zou, 2:43 PM), hap://blog.nmesunion mm/ubldhopping/igigo/new-albvry-resuamm�-inspection-grads-excellent-good-fair. 533 THE VALE LAW JOURNAL 12x:5]4 2012 County (OH)," Kern County (CA)'45 Maricopa County (AZ),a6 and San Bernadino County (CA)^' have implemented grading. Around this time, grading was proposed in Florida,48 New York State,99 Washington, D.C. so Albu uer ue 5' Chicago," E1 P s° 5a ss 9 9 � Paso,"' Kaufman T New ( da Haven, Pasadena,j° Pittsburgh," San Francisco,5° Alameda County (CA),59 aa. See Kaye Spector,Grading System for Cuyahoga County Restaurants Under Direction, PLALN DEALaR(Cleveland.Ohio),Feb.9,acro,http!//www.dmiand.mni/heAthfiVind".ssf/2oio /ox/gtadilLg_system_For_cnyahoga_m.html. 4s. See Jamas Burger,County Re"Aaa nu To Go Heahh Grades,BAKERSNEL.D CALILORNLaN,Oa. t8, soon, hnp.//www.bakersficldcaliforaiammn/AocalVxl393M3116/County-.e .r r.m-to -ger-health-grade. 46. See Duret Gilger,Mareaspa County's Resvmmnt f upatlon Prams Goa from Bang Eary on Ratau+arat-m Bang Elm Easier,PH%.NEW TaaEs:Oxo'BRO-a(Oct,17,sou,1.:16 AM), hap://blogs.phmEiixnewtimes.mm/bdla/sola/Lo/muimpa_romtty_implemenrs_vol.php. 47- See Imran Ghori,Coumy£awia To Cee Grada,PMSS-ENTERPRISE,Jnne s6,x004,ar B3, 48, See Miami-Dade Cray. Bd. of Cary. Commss, Res. No. R-195-07 (Ela 2007), http.//www,miamida&gov/gov tioL/4egismrfi[es/Min atmrs/y2oo7/o] 25=hl.pclf. as. See Jeff Klein,Coluvmer Pro'.Comm.,Restaurants Dun Are Erough To M.I.Yon Sick:An Analysis of Urumurary Conditions m Nes.York Ciry and WurJvster Come y Rrmaumna,N.Y. ST, SENATE (Aug. ss, 2005), hup,//www.nyse=m.gov/filw/nyss-migra,/KYCmd -Wemhes¢r-Counry-Resmmn -Enough-t make-You.pdf, se. Sm Jamie IL Liu,Mary Cheh Reintroduces Restaurant Hygiene Letter Grade Act,DQST(Jan. 19,stat,11:3o AM),httpe//dcist.mm/2oi1/o1/chel>_rerunsiden_leter-grades_for.php. 51 See Dan McKay, City Chaos on Grads for Eateries, AuJUOUEAgm J., Oct. s5, 2008, htlp://w 'Euljounlal.-rtVnews/meu'o/159462PM3newsmCMIO-15-o8.hmr. As described below,Albuquerque was already using one form ofgnd ng,but the city proposed to replace it with Icuer grade. s•. See Carolyn Walkup,Chimgo Pots Propose Pabst Hmkh-haperrion Grading System,NATmN's RRSrAORAN'r NEWS,Nov.30,1998,at J. s3 Sea Robert Seloer,Proposed Eatery Rating System Could Be Emy To Stomach,EL PASO TIMES, Mar.u,2003,at 1B. sa See Midrad Graham,Caunxit Opts To Bonuses sees Not R e kue,KanEIER,us Feb.26,aoo9, hapJ/www.kaufinanlurild.wMnews/article_Tfgdb96ybib-%3b- 841-z 8084761ee5.htm1. ss. See Paul Bats&Jargb Cohn, Mae 3 That Report Card In the Window?,NEW HAWN INDEP., June rd,sou,htcp://www.newhavemndepen&ncorglindex.php/archives/rntry/whars_din _mport_rasd_in_d1e_window. s6. See Gary Scow, Pasadena County Rgecv Letter C onda; Eateries To Post Health YNlatimu, PASADENA STM-NEWS,Oct n,2003,at A, n. See Patricia Sabunti,A-B-C Grading Srahedfor Restaurants in Allegheny County,PITTSBURGH POST-GAMTE,Mar. 30, sols, http://www.post-ga tte.com/swries/xLdonfionoife/a-b+ -grading-st2lled-foo-restauranu-in-allegheny-ronnt -3o934o. 584 FUDGING THE NUDGE AShland/Jackson County (OR)6' Huron County (OH)61 Kanawha County (WV),A`Santa Clara County (CA) ea and Ventura County (CA)." The Center for Science in the Public Interest advocates that"(sjtate and local governments should pass laws requiring the posting of inspection grade cards in the windows of all food establishmenes.x", Restaurant grading has its critics, however. New York City Council Speaker Christine Quinn called the City's grading system "inconsistent n66 and "borderline harmsmenti6j In March aorz,Speaker Quinn convened a raucous, six-hour oversight hearing and reported from a convenience sample that 66% of all restaurateurs (and 59% of restaurateurs who received 'A's) found the system"poor.n" Time magazine called the system"arbitrary and imperious."6l Sq. Sea Suvlme Heed, ,unman Hmtrh Ratings Win Approval, S.F. CHRON., May 12, zoo4, http://www.sfgare.comA,ealth/articie/San-Francisco-e staurinr-huldi-ratings -win-ay8oa6.p1hp. S9. See Meeting Norice: Community Meeting for a Proposed Food Facility Grading System,A DA COUNrr HEALTH CAIn 6'ERYIGES (Jat. 12, aorz),hrry J/www.azgov.org /aWdoamenWGradmgPmj.,Nmm,.pN. 6r. See Julie French, Voters Reject Restaurant Grading System,MArAND DAuY TIDINGS(Colo.), Nov. 6, zone, hapJ/�.dadod"tgs.m"Vapps/pbcs.dlVudcle?AID=/MO8no6/t E Soa /8a o60330- 61. See Cory Frolik Ruwumaa Grading Froposd Gen;an "F,"SANDUSU ASG.(Ohio),May a4, Zan,http://www.smduskymgismnm[n/udd42,802. 6a. So loci Kersey, Caryn Want, Rentrar.w' Health Repam 'Conspio.." CHARtFsro o GArETTE,Jan.x arra,hall)://wvgazcaacom/news/zoszomaooes 63. So MidwJle Guido,Enar,Grading Shit Debate,SAN JOSE MERCURY NEWS,Sept 19,arra,cit IB. A, See Ventura Cary. Grand Jury 2008-20r9, G Your Forem,Resraumru Ckon?,CountY or VEmurm Orme 23, x009), hap://portal.mnnrpfvenntra.org/pormVpagc/pottaVGmnd Jury/Reports/rAR499371/glsYonrFavrdmr stmmneCimn.pdf. 65. Sarah Klein &Camline Smith DeWad, Dirty Dining: Have Reservather ' You Will Now, CENIER FOR So. IN THE FUE. Ir,. 35 Untie 2008), hap://www.espiaetorg/dirtydiuing /DirryDiningEnportpdf. 66. Editorial, Food for Thougler, N.Y. DALLY NEws, Mar. u, aorz, IIUP.//arddes .nydzilynews.mm/2ol2-03-tt/news/31 4496_t_resrmrano-gmdu-lemtt gades-asmurant -inspection-process. 63. Michael Howard Saul,Quinn CrinialufRateamentGurdeSymen,WAu ST.J.,Dec.1y,2011, hnp://odine.wsj.min/uncle/SBmoo14a 052970204553904VII0302383916329a.hmd;Heard Around Town, cm& STATE, Mar. I, 2an2, htT.//www.tityandsramny.mnyhcard-rows -march-7-1012. 6a Glenn Collins, Rctimara n Voir,Anger over Health Inspersions, N.Y. TIMES: CM Room (Mar. r, 2012, 4:19 PM), hap://ciryrwm.bloge.nydmes.M./201z/rV0]/ratanrar M -void-angerover-health-inspectiolu; w Marc Beja, City Grilled over ''/nwaunaendes' in Rrseamam Grading System, AM N.Y. (Maez, on), hnp://www.amny.mm/ur6anire 585 THE YALE IAVJ JOURNAL ua:57q aou The Wall Street Journal and New York Times documented.suggestive evidence of gaming of the grading thresholds, which we use as a starting point of our analysis of New York below-" Despite the pivotal role that grading commands in the debate over information disclosure (and the exhaustive review of the literature by Fung, Graham,and Weil), restaurant grading's merits tum out to rest on remarkably fragile empirical grounds. The only large-scale empirical study of grading olamines Los Angeles around 1997." To cure this empirical gap, this Article amasses large-scale microdata from over 700,000 restaurant inspections in ten other jurisdictions to evaluate the efficacy of restaurant grading.l' For expositional simplicity, the analysis of our research team focuses on San Diego and New York, but the findings generalize to the other jurisdictions." We show that the benefits of grading are vastly overstated, and costs vastly understated. The regulatory design, implementation, and practice in these jurisdictions are flawed at their core.As practiced, regulators fudge the nudge. The findings, in brief, are fourfold. First, nearly every restaurant in San Diego receives an 'A,' limiting the meaningfulness of grades. Second, New -1.81z 39/city-grilled-over-inromistauiu-in-rumumnbgmding-syuem-x.3586988; Press Mease,N.Y.C.Council,Majority of Restaurant hhsp etiou Survey Participants Remixed A Grades and SNI Rated the System Poorly (Mar. 7, 2012), http://councd.nyc.gov /htmf/releases/pdfs/restamerneksm.pdf(reporting results from a survey asking, among otter questions, how would you race the letter grading system?"). The author of this Article testified as an independent expert at this hearing.See Empirked Fads About Reeteumnt Grading,Hearing Before the N.Y.C. Council Comm, on Govemmemal Ope aiom, Comm. on Hea sh, Comm. on Oversight & Invrsnganam, & Comm. an Sinal! Bur. (Mar. 7, 2012) (sentiment of Daniel E. He) (on file widx author);see abe Daniel E. Ho,Op-Ed.,Improve Resm7am a Report Gals, N.Y. Times, Mat. 7, amu, http://www.nydmm.wm/20xx/03/07 /opinior�/dre-ruuuraw-grade-system-is-brolwn.htud. 69. Josh Owreky, Gioirg an F no New York}Itrsmurwu Grading System, TIME,July 20, 2010, http://www.dme.mWtime/ ation/utide/o,8599.2005191,00.html. m See Smnathi Reddy&H0I.S hdl.,,Many Eatery High Marks Are Close Call,WALL Sr.J., July a8, aun, http:Nothme.wsj.rom/azticlrJSBxo0o4r4o53m90688830457647x3x36645f488 .had; Brian J. McCabe, Grading New York Barawmao: Whats in An A'x, N.Y. 11MEs: FIVETH1areErGHT (Jan_ 19. 2011, 8:33 PM), llnpJ/frvMturyeigl¢blogs.nyomes.roMxou /oVi9/grading-new-york-mmutana-wham-of-m-a("Closer inspection of the underlying data reveals a suspicions distribution ofwma aria rear the cutoff point between an A and a B.'). V. See FUNC 6T At.,supra ame LJ, at 78, 82-83. 193-94 (mlying exclusively on studies of Cos Angeles in assessing the efficacy of restaurant grading). 7a Fie of the jurisdictions grade:San Diego,New York,North Carolina,South Carolina,and Wuisville.Three of these jurisdictions scow,but do not grade:EI Paso,Austin,and Seattle. hvo jurisdictions d0 not scow or grade: Chicago and Florida. Nongradi ng jurisdictions provide a comparison group,which confirms the effects of grading.See infra Appendix D. 73. See infra Appendix D. 586 FUDGING THE NUDGE York grades vary widely, but, unlike San Diego's underlying numerical stores, New York scores exhibit little substantive consistency. A score (or grade) in one year predicts little about the restaurant's cleanliness down the road.Third, differing inspection criteria provide one compelling explanation for the difference in consistency between San Diego and New York. The relative complexity of New York's inspection criteria appears to impede uniform scaring across inspectors and restaurants. Fourth, grading in New York has had no discernible health benefits but may come at a large, previously unrecognized cost in administrative resources. Specifically, grading reallocates inspection resources away from restaurants that pose the great t public health risk toward grade resolution at lower-risk restaurants. These findings speak richly to longstanding puzzles in regulation and administrative law. How should policymakers best channel administrative discretion? How does the institutional design of inspection or disclosure regimes affect regulatory outcomes?How can we disclose information to enlist private actors to property incentivize regulated industries?The concrete policy implications are considerable. Targeted Transparency's emphasis on simplification shouldn't just apply to information disclosure, but also to information collection. What proponents of targeted transparency and grade reformers miss is that cognitive limitations impede not just users of information, but suppliers as well. At the same time, all raw microdata underlying the letter grades (i.e., the full inspection database) should be made available in machine-readable format. Combining simple retail disclosure (letter grades) with wholesale complex disclosure (microdata) empowers information intermediaries to develop better and more useful information summaries. More generally, the findings show that targeted Transparency is extraordinarily sensitive to context and regulatory design. If targeted transparency teaches us to target information, this study shows that targeting can be achieved in myriad ways that undermine the effectiveness of disclosure. The Article proceeds as follows. Part I examines the historical antecedents of current grading regimes and the extant evidence of grading's benefits. Contrary to the conventional perception of Targeted transparency as a phenomenon of the last twenty years, restaurant grading was a common practice in the 19409. Its demise was tied to deep skepticism in the public health field about the benefits of grading. Currently, the only systematic empirical evidence in support of restaurant grades comes from a study of Los Angeles around 1997. Part lI articulates the empirical approach of this Article, namely to examine evidence from other jurisdictions. It shows why credible policy evaluation of the impact of restaurant grading is riddled with challenges. Examining major metropolitan areas, we show that institutional features of inspections and of grading vary dramatically across jurisdictions, providing 587 THE YALE LAW JOURNAL I22:S74 aoia good reason to doubt that the benefits of grading in Los Angeles generalize to other jurisdictions. Part III turns to the evidence in San Diego, which has been practicing restaurant grading since 1947. Part IV discusses the evidence in New York, which adopted grading in July 2010. Part V examines whether the complexity of inspection criteria may explain the divergence in consistency of scoring between San Diego and New York. Part VI finds no evidence of the intended health benefits, but documents that New York's implementation comes at a previously unrecognized cost of shifting inspection resources away from the highest-risk restaurants toward grade resolution. Put VII discusses policy implications. I. THE LANDSCAPE OF GRADE REFORM A.Hutorval Antecedents In 1934, the National Recovery Administrator proposed a Code of Fair Competition for the Restaurant Industry.J4 Although the Code Authority, which had delegated the task of developing minimum sanitation standards to a committee,''would fall by the wayside;A the Public Health Service(PHS)and later the Food and Drug Administration (FDA) continued to develop a model fund Code, first proposing grading in 1940.°1 Drawing on letter grading for milk,j6 the model code proposed rating restaurants with letter grades for M. Nail Recovery Admin., No. 28a, Cade of Fair Competition for the Resrawant Industry (Feb.to,19M). rS Id.-514. aB. Sw ALA.Schechter Poultry Corp.Y. United States,295 V.S.495(1935) (invalidating the National Industrial Reeovery Act Bud die usociated industrial codes). T). U.S. PUB. HEALTH SERV., ORDINANCE AND CODE REGULATING EATING ANn DRINxING ESTABUSHHEMB(194o)(hereinafter 1940 CODE].Grading also appeared in the model food rode in 1938, but this version was "renradve" beczuu it had not been reviewed by die Saturation Advisory Board. See U.S. PUB. HEALTH SERV., ORDINANCE AND CODE REGULATING EATING AND DRINKING ESIABLISHMEMS 2 (1938). These model codes are advisory and not codified by the federal govermnent. 0. Su RICHMD H.BOEHNKE,INTERNATIONAL SURVEY ON PUBLIC POSrING OF RESTAURANT INSPECTION REFORTS, AND/OR GRADE CARD POSTING SCHEMES BASED UPON HEALTH INSPECr1ONS 4 (..Be) ('A decade lacer, in 1934, die sone United States Public Health Service introduced due first model food code. h was based direedy upon the cxisung 1924 Mitk Code complex with the milk horde letter grade system."). 588 FUDGING THE NUDGE sanitation standards." A restaurant was deemed grade 'A' if it complied with each of seventeen inspection items (ranging from standards for doors and windows to refrigeration of perishable food)."Grade restaurants complied with most items but violated one of five (evidently less egregious) specific items (i.e., floors, walls and ceilings, lighting, ventilation, miscellaneous). Grade'C'restaurants failed to meet either standard."Modern ideas of targeted transparency were already apparent. The code required restaurants to display grades in a fashion readily visible to customers" (not unlitce the National Recovery Administration's Blue Eagle"). As one PHS official described it, requiring "public display of a [sanitation] grade notice inall restaurants' would exert"competitive effect.. . to improve...sanitation."s s The idea proved popular. Large cities such as St. Louis,86 Atlanta,s' San Diego 8B and Pittsburgh" instituted grading systems in the t94os and tg5os.90 By one estimate,roughly four hundred U.S.cities had grading systems in place in 1951.9' PHS and FDA revised the model food code over the decades."In 1962,the 78. The model code included versions for"grading'atd'non-gradurg'jurisdictions.Sea 1940 CODE,:uym note Tl.at 5It, Be. Id.at ry-31. a,. Id at 31-3z. sa Id at 32. y. Id-1 14-15 84 Sia DAND M.KENNEDY,FRFIDOM FROM FEAR:THE AMERICAN PEOPLE IN DEPRESPON AND WAR,1929-1945.at 183-84(2001)(describing the Blue Bagle). 8, AW. Fuchs, The U.S. Public Health Service Ramurant Sanitation Progrmn, 3a AM.J.PUB. HE.,UTH 848,850(tgga)(paraphrasing language from the model code). 86. See Maurice E.Trout,Cleaning Up d,e ltesraurams,38 NAT't MON.Into.335.335(1949) BT. See Cuumil To Comfier Re nnoman Clean-Up,ATu A CONST.,Sept.16,gg45,at 3A;Grade "C'Eating Plato Have 3o Days To Improve,AT A DNLY WORLD,Oct.a9,11,46,at 6. Bs. See Brooke Williams & Agustin Armendam, A Reope for Trouble: Coming Clean on liestaurma Grades,SAN DIEGO UMON-Two.,July ax,2007,htrp://aww.signonsandiego.mm /unionnib/20070722/news_lunazro ipe.html. as. See Pimbrngh Forcing Cafes To Clean Up or Close Up,LA TIMES,Ma.6,1951.at Be. 80. Washington, D.C. proposed grading in 1943. See Mne Hagnce, BBI Tighten Restaurant Sanitary Code,WASH.POST,SepL 15.1943.at 1o. 8,. See Pim6urgh Forcing Cafes To Clean Up or Close Up,.supm note 89,at 88. 4a. See Food Service Sanitation: Proposed Uniform Requiremaars for Sett and Local Regulatory Agencies, 39 Fed. Reg. 35438 (Oct 1, 19]4), uie6droum by Food Service Sanitation:Withdrawal of Proposal and Termination of Ride Making Proceeding,42 Fed. Reg. 15,428 (Mar. z2, 19T/); see also Clinton L. Rappole, Sanitation in the Food Service 589 THE YALE LAW JOURNAL 122:574 2013 model code continued with letter grading (and required posting of grade placards), but based the grades on the total number of"demerit"points issued for each violation.93 In 1976, after a failed attempt to promulgate federal uniform sanitation standards,99 the FDA abandoned restaurant grading altogether. In place of demerit points, the model code proposed a loo-point scoring system,with weighted points ranging from 1 to 5 assigned to forty-four violations.9s A score below on required the restaurant to tante corrective action within forty-eight hours, and failure to do so would potentially lead to a shutdown.g6 Although the model code required that the inspection report be available to the public on request,91 it made no mention of publicly posting the score or any other inspection output. A contemporaneous report by the General Accounting Office°a reviewing sanitation inspection systems, explained that "[p]ublicizing restaurant inspection results [was] surrounded by some controversy" among health offrcials.99 FDA officials acknowledged the benefits of disclosure (empowering customers and incentivizing restaurateurs), but noted numerous criticisms of publicizing results, in particular the limited consumer understanding of inspection results and the fact that"[c]onditions found on inspection date may change greatly (degrade or improve) on later days," thereby "giv[ing] the customer a false sense of security.""' By 1993, the FDA had abandoned numerical scoring entirely."' Currently, the FDA acknowledges that "scoring Operation: The Implimriuw of the Proposed S..4rainn Ordinance, r8 CoRNEu HOTEL & RESTAURANTADMIN.Q31(r9u)(describingproposed modifiwdons to dee food code). 93. U.S.DEPT OF HEALTH,EDUC.,&WELFARE,PUB.HEALTH SERV.,FOOD SERv[CE SANITATION MANUAL 77 (1962) (proposing a model ordinan¢ for dre grading of ftrod service establislunens). 90. U.S. DEVT OF HEALTH,EDUc., &WELFARE, PUB. HEALTH SERV.,FOOD&DRUG ADMIN., FWD SEAQ 3CESANUAT ON MANUAL 74(1976). 9s. See id.at 88. 96. Id.at 75. 97. Id.at 74. 9S. The name -Genet Accounting Office' was lunged to die "Govmunem Accountability Office"by the GAO Human Capirzt Reform Act ofroo4,Pub.L.ro8-z7L,5 8,US Stat.8u, 8r4(codified as a note in 3L U.S.C.5 7oz(2006)). 99. COMPrEorUFR GEN. OF THE U.S., GEN. ACCOUNTING OFFICE, MWD-76-43, FEDERAL SUPPORT FOR RESTAORANr SANITATION FOUND LIRGELY INEFFEctnt 19(1975) loo. Id.at m. rm. See U.S. DEVT OF HEALTH Be HUMAN SERYS.,PUB.HEALTH SERV.,PWD&DRUG AOMUL, FWD CODE,ANNEX 4,at 38-4r(t993)(L<mmmending comparison ofrsrablishlwrts by the umber of critical viBlatims, but Boring chat"there is w defined point at which a score translates Hiro a significant health ha2aad"and doing away wirh the prior too-point score). 590 FUDGING THE NUDGE may provide a mechanism for consumers to make informed choices," but also points to "negative consequences" such as a restaurant receiving "a high numerical or letter score while exhibiting some very serious deficiencies."1O In sum, while scholars of targeted transparency may be right that such regulatory interventions are a phenomenon of the last twenty years," the historical trend in sanitation inspections is,if anything,the reverse. In line with grading's demise in the food code, most local jurisdictions abandoned such 114 systems over the course of the twentieth century. B.Public Health Doubts The demise of the first generation of grading schemes reflects a deeper skepticism in the public health community. A crucial predicate for restaurant grading is that there are "consensus metrics"'°s: established methods to consistently measure attributes of direct interest. For instance, if standardized tests represent a consensus metric (i.e., measure attributes of direct interest), "teaching to the test" may not be problematic. As one administrative law casebook writes: "It is useful to provide information . . . about restaurant cleanliness, because most people agree on the relevance of those factors and how to measure them.""' A review of the public health literature, however, reveals that such consensus may be illusory: put simply, "a single indicator has not been developed that summaries all the relevant factors into one measure of safety.11O7 Several obstacles impede consensus. First, sanitation conditions can change rapidly. Even within a single day, an inspection during the lunch rush may yield sharply different results than in the late afternoon."[T]hc traditional inspection represents a snapshot of the facility operation,or about one hour of time from what may be an 18-hour day. . . ." s Moreover,restaurant patrons, staff, and management—and, in turn, sanitation conditions—fluctuate rapidly +Oi U.S.PUB.HE mSERV.,FOOD&DRUGADMIN.,FWD CODE 569(2009). 101. Se, e.g..FunG er M_,supra note 1;at 20("In die last twenty years,targeted transparency policies have played apmminent role.../). +oa. Sm BOEHN.,n,pm note 7a,at4;Pius6n+gh Furring Cnfes To Clmn up or Cbse Vp,supra note 89,ar Ba. AOMINISTMTIVE TowAND REGUIATORY POLICY:Pkmts S,TExT,AND GASES 740(Stephen G.Breyer et al.eds.,7di ed.lou)(discussing FLING erns.,supra note 12). os. Id. +o7. Chris J.Wlant,Scam,Grader,and Commnnimsing About Foad Safery,61 J.ENvrr.Hassm 37, 38 0999) ,os. Id.at 37. 59t THE VALE LAW JOURNAL 1¢x:5]4 xonx across days." Second, inspectors may use a seemingly objective scoring rubric in drastically divergent ways. As two environmental health scholars put it: "[I]f two professionals evaluate the same restaurant at the same time . . . are their grades the same? . . . If one professional inspects the same restaurant at different times. . . will the grades be the same?""o Assessing the existence and severity of violations, such as an "improperly constructed" surface,"' "inadequate" "personal cleanliness,""' or food not in "good condition,""a necessarily requires inspector discretion and is thereby subject to variability in implementation across inspectors. As another health official notes, "many departments have extensive checklists [but] it is the norm for every single person to do an inspection differently."'"One study of Tennessee from 1993 to x000 documented that mean scores (out of a stale of too) for igo inspectors who each performed at least too inspections ranged from 69 to 92."s Inspector heterogeneity leads"restaurant inspections[to be] inherently inconsistent."" Third, consumers may misunderstand the import of a disclosed grade (or score). A random phone survey of two thousand Tennessee adults, for example, documented highly unrealistic expectations of the inspection system. Over 5o% of respondents believed health inspections should be performed at least twelve times per year;Tennessee in fact conducted two.Forty-five percent of respondents indicated that the minimum acceptable score to eat at a restaurant would be 9o; the mean score in fact was 82.11' In addition, +os. Cf.BOEHNxE,supra note 70,at a5("Given the extremely high turnover race in die restaurant industry,both in s nffand in operamrs,a stated grade value Feared at a restauraut at the ame ofdae patrom'visits is much more likely to be dated history dun currently accuna."). sm. Owen H.Sewer&Thomas H.Hatfield, Gmdutg S)stmv for Aemil Foal Purifier.:A Risk- BemadAndpis,631 ENVPL,HEALTH ca,H(a000): ni. Bureau of Food Safety & Cmry. Swimmer, Self-lmyersiun Worksheet for Pond Serves EsurG/uhmena,N.Y.C.DEe'ToFHFn1.TH&MENTALHyaJFMa(Dec.xmo),http://...nycgov /luml/doh/downloads/pdf/riVself-inspection-workshe t.pdf(Violation sC). ax. Id.at a(Violation 6A). 111. Food & Hous. Div., Retail Fcod Facility Operamr's Guide, COUNTY OF SAN Deco DEPT OF ENvn.. HEALTH 6 (2007), htcp://www.sdmu ty.m.gov/deb/food/pdf /publiestions_opguide.pdf[hereinafter Retail Food Facility Opeavar's Guide](Violation 13). rµ THOMAS PEAcLacK,Is IT SAFE TO EAT OUT?How OUR IAGt HEALTH OPFICW,s U,smE RESTAm,NIYTOAs5LAEFOOD SAFETY...ORDOTNEY?59(xoox). 11s. Timothy F. Jones et al., Restaumm,Impaction Sam and Foodbame Disease, m EMERGING INFECnOUS DISEAEEs 688,688(x004). n6. Sewer&HaKeld,supranoteuo,at a5. .p. Timothy F.Jona& Karen Grimm, Public Knowledge and Attitudes Regarding Public Health 'MFx ioru ofPvtauraru,34AM.J.PREVENFNE MED.sno,SII-Ix(x008). S9t FUDGING THE NUDGE consumers exhibit inconsistent risk perceptions of grades and scores,subject to considerable framing effects: "A single grades fails to deliver a consistent message regardless of the underlying purpose. A survey of college students and food safety professionals showed that while grade signs affected willingness to eat at an establishment, there was no consensus on the meaning of a grade or score. Most tellingly, a sample of seventy-two food safety professionals was asked to interpret a 'C' grade. Forty percent said the restaurant was "average,' 32% said that the restaurant had problems,and the res[were unclear about the meaning.'''Only zx%of students would be willing to eat a a'C'-graded restaurant(one student thought'C'stood for "compliant"), compared in 65"/ of food safety professionals." To better understand inspection Scores,one New York Times food reporter invited a health inspector to score his home kitchen.A New York restaurant receives an`A' if it scores below 14 (violation) points. The score for the reporter's kitchen was 77."' Not only would a score o£77 knock the kitchen out of the`A'range,but it would also put it at serious risk for an immediate shutdown. Most home kitchens would arguably fare poorly. Violations such as washing hands in a sink where dishes are done or failing to label food in Tupperware containers are not necessarily what consumers perceive as salient health risks. Fourth, grading is in some tension with evolving conceptions of food safety. Since 1993, the FDA has advocated so-called Hazard Analysis and Critical Control Points (HACCp) principles.''The chief idea of HACCP is to shift inspections away from"floor-walls-ceilings"approaches (where violations at endpoints are counted) towwd a focus on preventing structural risk factors in the process of food preparation.'�3 Restaurateurs should focus on critical us. Thomas H.HatGeld&Owen H.Sewer,Preference Reaernals in Grading Sysnmufor Retail Food Fadrtfa,63 J-ENVFL,HEALTH 19,23(W00' nq. Lauren Dandes&Sushama Rappaksa,Seuna and Coal.:A Sampling of Hoa College Srudrnrs and Food Safety Nvfesriowl.Interpret Runanno,Inspersion Ruulu,64 J.ENvrL.HEN.TH 14, 1s-16(ac o ). no. Id.at 16. m. Henry Alford, Would the Ciry Sian Dolon Your Koolau?, N.Y. TIMES, Sept. aa, amo, hop://www.nytimes.colnlaoto/o9/a9/dining/a9inspxmnhunl. a U.S.DEFT OF HELLTH&HUMAN SERVS.,PUB.HFALTH SERV.,FOOD&DRUG ADMIN.,FWD a CODE, IFITRO. (1993) ("(T)he new Code incorpon¢s a framework for du application of Haard Analysis Critical Control Point(HACCP)principles a Tenor...'). nes. BOEHNKE,Supm note 78,at a4:nee FOOD RES.INST.,FOOD SAFETY 1994,at 470(1994):CM for Food Safety & Applied Nutrition, Managing Food Safrry: A Regulators Manual for Applying HACCP Pnmiples nes Risk-Based Renal and Food Sovire Ivpeetiaru and Ernluaang Voluntary Food Saffty Management S]nnenu,U.S.DEFT OF FDALTH&HUM.SERVICES,FOOD& 593 THE YALE LAW JOURNAL 122:574 I.Ir points of the food preparation process, when food may be contaminated or adulterated—e.g., delivery, preparation, and serving." Because more of the onus is on restaurants to establish documentary records of the flow of food, some argue that a more educational approach to inspections is required."[Tlhe application of HACCP principles; in that sense, "is not conducive to simple grading schemes,""s which penalize violations without necessarily addressing the processes that lead to those violations.Grading can also erode the ability of inspectors to educate by setting up an antagonistic relationship with restamateurs.'e6 The result of these obstacles, as we show in Section IIB, is that although virtually all jurisdictions follow the FDA's model food code, there is little uniformity in the way violations are assessed.Nine of twenty top metropolitan areas do not use any formal numerical score at all.And in the jurisdictions that do use numerical scoring, violations receive drastically different weights. Numerous health practitioners have proposed alternative scoring and weighting techniques, arguing that extant techniques are deficient." One study synthesizing "best practices," based on a survey of forty-seven state and local inspection systems,never mentioned scoring or grading as relevant practices."' Perhaps the most compelling synthesis of these critiques comes from Richard Boehnke, who surveyed forty-five state health agency senior officials DRUG ADMIN. I-2(Apr.amm),hap://wwsv.flagov/dowTdoad/FwQ/FoodSafety/RemIF.d PmtmdofManagingF dSaferyHACCPPrinciplesfR gulawrs/UCMo78L59.pdf. UT See Ctr for Food Safety&Applied Nutrition,supm Tore h23,at 1-2. i. as. BOEHMtE,supm non 78,atM;scala LORAJULDUSrt&DOUOvs Ro9EltTBaowN,FIACCP& SANITATION IN RESTAURANTS AND FWD SERVILE OPERATIONS:A PMLTLGL GUIDE BASED ON THE FDA FWD CDDE 155(2005) ("Traditional inspection is relatively resource irmoave and inefficient and is reas:dw rather than preventive compared to the HACCP approach....."). .26. See Paid FnuNtin,Holt Deponerms Berwning Educators,NAmorv's REernut oor lams,May 6, 2oo7, htip://iun.coni/ardde/Itegd deparmiaim-b=ming-edti=n ('In New York, a reTvr of'nsverars dum'seems to pervade much of de namorani cremmmy..."). "r. See, e.g., Brian Emanuel, Grading a Food Pnablishmem, 58 J. ENVTL. HEKTTx 20 (1995) (proposing a new scoring system);David Z.McSwane et al.,In Search of the Ingredients ofa Saoessful Recoil Food Compliance Program,50 J.BNvM.HEALTH 342,3 (1988) (discussing crimisms by saniution offimre of 1976 food code sroring); Robert K Seevemon,A Food Service Evabhuhmem Evaluation Program Prvedme for the sy8oi and 1y9o's, So J. ENvr.. HEALTH a5(1987)(proposing scoring based on ha2ard potential). na. See McSwam et al.,supm Tole it .But see Kaddeen Irwin et al.,Itrsula efRourim JZOAmmry Ims,euiam Can Pndfet Owb.mks of Foodbona Nass: The Swde-King County E2perieae,79 Aea.J.PUB.HEALTEL 586(1989) (using case-camel matching with a sample of eighty-four au ...m re show that poor inspeores ction swere associated with a f dbonit illntss o rtrabreak). 594 FUDGING THE NUDGE across the United States." Health officials pointed to each of the criticisms above,concluding that"consistent inspection standards are never achieved.""a In addition, officials noted two more nuanced points. Restaurants may take measures solely to achieve a high grade, which may not in fact reduce the risk of foodborne illness."' Conversely, officials noted the presence of "political pressure . . . inevimbl[yJ to raise grades" and that "all grades go toward 'A' through pressure on inspectors...s' Pressure can be quite direct. In 1988, twenty-eight of seventy New York inspectors were charged with extortion,"' and in Los Angeles, one inspector was caught on camera saying, 'It's going to Cost$zoo to get an'A.­" In light of these critiques, many have Called for studying the efficacy, of disclosure systems."'Yet as of z000, only one jurisdiction (Florida) reported ever having evaluated any form of grading.y6 C.Los Angeles Faith So where does the enthusiasm for grading come from?The only credible, systematic,empirical evidence for the benefits of grading comes from one set of studies focusing on the implementation of sanitation grading in Los Angeles in January 1998.'17 Prior to January 1998, Los Angeles scored restaurants numerically based on the(weighted) number of violations,with a score of too indicating no violations and a score of o indicating full noncompliance. In December 1997, the county proposed that these numerical scores be summarized and posted as letter grades. Scores of 90-mo would tum to 'A's; ug. See BoeHNe6,supra now 78,at z. ,yes. Ld.at u. .y. hL at rz. ,ys. See Howard Kuttz,28 New yank City Lteswuram lmpeetors Atoned oflixtmtion,WMR POST, Mac x5,r988,at A8. 134. Health Irnpection Bribe ;z pori Prompts Nadine, LA. Muss, Nov. u, 1998, hap://ax ies.latimes.mnVx998/nov/iplocaVme-4x653. us. See K iic Filion&Eunglas A.puwdl,The Use ofWsranrant Lupeetion Dedusure Systems as a Meares of Communicating Food Safety Infonrw.tian, xo J.FOODSERWa e87(2009); Seiver& Hadfield,supra now sro;Wiana,RTM now 107. q6. See Bomt*CE,Supra note 78,at i- 137. Studies of other inspxdon syscems exist.but dww ate wt focused on the impact of grato ding per se. See, e.g., Sylvaznrs Thompson, Ron de Binger se Olayemi Karin, tmpe¢bn and Dvdosure System: A Case Srudy, sal Ben. FOOD J. 140 (xoo5) (assessing Toronto's food irupecU0u systnm). 595 THE YALE LAW JOURNAL ux:374 anti 80-89 to 'B's; and 70-79 to 'C's. Scores below 70 would be posted without a letter grade. While Ginger Jin and Phillip Leslie published several leading papers based on Las Angeles, the core findings—cited widely by proponents of grading"'—are from an article in the Quarterly Journal ofEcarom cs.19 The article examined data from 1996 to 1998, encompassing (1) restaurant inspections in Los Angeles County, (z) quarterly sales-tax data for 57%of these restaurants, and (3) admissions to hospitals for food-related and non-Eood- related digestive disorders for three-digit ZIP codes."" Assuming that the adoption of grading (by the county and individual cities'41) was exogenous, the article examined several effects of grading. First, mandatory grading increased numerical inspection scores by 4.4 points.'42 Roughly z3% of the variation in scores appeared In, be explained by restaurant-specific attributes,"n suggesting that the score in one year reveals meaningful information about the restaurants future cleanliness. The "effects on hygiene from the grade cards [we]re realized fairly rapidly,""4 within one year of the introduction of grade cards. Second, mandatory posting caused statistically significant changes in revenue of (a)a 5.7%increase for'A'-grade restaurants, (2) a o.70/ increase for ns. See,e.g.,FLING ET AL.,cup.note ua,at 0,(discussing studies by Jin and Leslie and noting that those "found significant decresus in food-borne-illness hospidi2ztimu"); Bd. of Health, Notice 'Adaption ofAmadmenu to Article gr of the Nem Park Gty H." Code, N.Y.C. DEBT OF HEN.IH & MENrA HYGIENE (2mo), hap://www.nyc.govAi=Vdoh /dowNoads/pdf/not4z/2oio/Ardcle-gi.pdf(noting a w20% deduce in hospitaliutions for food-borne illnesses" m justify The adoption of restaurant grading in New York); Klein& DeWaal,supra note 65.at 32(advoeating that all jurisdictions adopt maintain grading and noting drat "the grading system has contributed to a ao percent doresse in foods omeo- illness-related-hospiuGution"in Ices Angeles). li ng. Su Jin&Leslie,supra now 3g see also Ginger Zhe Jin&Phillip Leslie,Repuratioual Ineeraion for Retention, Hygume, I AM.ECON.J.: M CROECON. 237(so09) (estamnning las Angeles evidence in support of me.t grading); Paid A Simon et A, Impact of Ramumm Hygiene Grade Cards on Foodborne-Dismu Hospimlision.in Los Aege4s Couruy,67 J.ENvrt. HP V.TH 32(2005)(same);Ginger Ste Jin&Phillip Leslie,The Care in Support ofResmurant Hygiene Grade Cards,CHOICES,ad Quarter 2005,2t 97(same). Mo. Throe-digit ZIP codes are simply dee first fine digits of conventional ZIP codes, and are thus more highly aggregated. Mr. See Josh Meyer, Loophole Hampers Rumurwrt Cmddown, L.A. TIMES, Feb. 4, 1998, hap://arcades.lati=s.wnViggg/feb/o4AmaVmc-i 9> . I. 142. See Jin&Leslie,copra note 34.at 44-25&tbl.3. Mr Id.-1424&IN.3(reporting an R'of o.5874 with restaurant fed dfccs and an R'of 03574 without feed effete,To a panel regression of sanitation am=).R'"measures the proportion of die cod variation(in a variable]that is explained by the fitted[regression]line."GEORGE CASELIA&ROGER L.BERGEN,$TATimca .INFERENCE 524(Iggo). 144. Jin&Leslie,raps note 34,at 426. 596 FUDGING THE NUDGE W-grade restaurants,and(3) a I%decrease for'C'-grade resmurams.`n Third, the study found some evidence that grading affected inspector discretion, particularly a spike of the use of the score go, the cutoff for an W.'4' Fourth, comparing hospitalization rates from 1995 to iggg for food-related (and non- food-related) illnesses between Los Angeles County and the rest of California, the study found that mandatory posting "cause[d] a highly statistically significant zo percent decrease in hospitalizations."'°r Although the study performed a sophisticated decomposition of the health effect, the intuition of the effect, as the article discusses, ran be seen in Figure 1 (adapted from Table V by Jin and Leslie's study). Figure I. INTUITION OF LOS ANGELES GRADING EFFECT ON HOSPITALIZATIONS FOOD-NCLATCD NON"'FOOD-IIELATED R. Of °a Califomia Gxding hcgins cg o Los Angeles e �o T1995 1997 1999 1995 1997 1999 Year Year Hospital admissimu for food-[elated digestive disordcts on the left panel and for non- food-related digauve disorders on the right p nel for IOs Angeles County(in black) and the rest of Califonda(in gray).These data are adapted from Table V of Jin and I lies study and illustrate the gist of the panel design.Food-related hospimlisadons drop slightly more sharply in Ins imgeles than in the rest of Caldonua, while non- food-rdared hospialbatime increase more sharply in die rest of California. The left panel plots the number of hospital admissions for food-related illnesses (on the y-axis) against years (on the x-axis). Los Angeles County Is 14s. Id.at 4a9&tblq. 14s. Id.-1 133-1 147. Id.at439-40 597 THE YALE LAW JOURNAL 123:574 3012 plotted in black, and the rest of California is plotted in gray. In 1997, for example, 405 persons were admitted for food-related illnesses in Los Angeles, compared to 654 in the rest of California The vertical gray line indicates the beginning of restaurant grading. The right panel similarly plots non-food- related hospitalizations for digestive disorder. Hospitalimtions for food-related illnesses dropped in Los Angeles, from 405 cases in 1997 to 351 cases in 1998, which is not the case for the rest of California (nor did non-food-related cases exhibit such sharp shifts).As Jin and Leslie state,"This is basic and compelling evidence in favor of hygiene grade cards causing an improvement in actual health outcomes.--a The Jin and Leslie study is admirable. It compiles rich microdata from multiple sources, examines specific mechanisms by which disclosure affects restaurants, and applies modem econometric (panel) approaches to study the effect of grading. There are, however, reasons to question the findings on Foodborne illness. The number of food-related hospitalizations is very small (certainly relative to the population incidence of food poisoning) and likely subject to sharp movements in such a short time frame(e.g., the sharp increase in 1996 for California). Los Angeles's drop in food-related illnesses possibly began before the imposition of grading. As one Journal of Economic Literature review points out, "national trends indicated a reduction in foodbome illnesses (and hospitalizations) during the same period that the grade cards were introduced in Los Angeles County."t'"Los Angeles and the rest of California diverge in non-food-related illnesses, suggesting that the rest of California (or non-food-related illnesses)may not be a good comparison group."' ya. Id.at 438. 149. Clifford Wirssion,The Efficacy offnfomiation Policy:ARadew ofArdwn Fung,Mary Graham, and IMvid Wal's Frill Disclosure: The Perils and Promise of Transparency, 46 J. EcoN. LorsATURE 704,709(loa8).Wmsmn also points to attention paid to the E.WE Ombmak and the 1998 adoption by the USDA of11ACCP testing.See id.at 709-11. +5o. The credibility of a"differemc-bi-ddlerentn"design binges on die compmali iry of a long preintervention time series.The Jin and Leslie linear least squares regression model captains elle outcome of die number of hospital admissions for digestive disorders in one of fifty- seven three-digit ZIP codes for a mondi(logged)with(t)fixed effects for each ZIP code aid indicator for whether the illness is food-related/non-food-related, (a) fixed effects for months,(3) the proportion of a ZIP code subjax to mandatory grade posting(proportion mandamry), (4) the proportion of die UP code subject to voluntary grade posting (i.e., when a city council has not made posting re ndamry) (proportion voluntary), (5) an interaction effett between an indicator for whedree die illness is food-related(food-relared indicator) and die proportion mandatory,aid(6) an interaction effect between the food- related indicator and die proportion voluntary.The joint effects of the coefficients for(R) and (4)provide the na effect estimate of a ao%reduction in Food-relared hospitalira ions due m maidamry posting.SeeJin&Leslie,supra note 3q,ar 43810&tbl.6. 598 FUDGING THE NUDGE while understanding such threats to validity is important(and could be the subject of an important replication study), we do not focus on these for the remainder of this Article. Instead, this Article shows that the evidence in other jurisdictions—which have never been studied before—should give pause to the unfettered enthusiasm for restaurant grading, and that targeted transparency should focus to a much greater extent on institutional design. H. EMPIRICALLY ASSESSING GRADING A. The Confounding Nature of Grade Reform To assess the causal effect of grade reform, the ideal experiment would randomize a large number of jurisdictions (or restaurants) to be subject to sanitation grading. Randomization would ensure that jurisdictions (or restaurants) subject to grading are comparable to those that are not."' When researchers cannot control the intervention, observational studies aim to replicate that hypothetical experimental template by finding Units that are similar in all respects except for restaurant grading. Herein lies the basic challenge for policy evaluation. Restaurant grading is essentially never randomly adopted. To the contrary, as with much regulatory reform, the intervention is,first,often a political response to a perceived crisis,and second, rarely proposed in isolation."' The former means that any purported benefit may be due solely to regression to the mean alone. Pasco County, Florida, for example, instituted a grading system in the 199os after a severe outbreak of foodborne illness."' A reduction in foodborne illness after the imposition of grading may simply reflect a return m the pre-outbreak risk. The latter (that grading is usually pan of a reform package) means that isolating the causal effect of grading becomes extraordinarily difficult In Los Angeles, for example, grade reform was a response to a series of rs,. Of worse, any single o ndombation might still result in groups drat arc distinct, but randomization over a large sample of jurisdictions guarantees dot du chances of such imbalances are vanishingly small. rn. Cf. Daniel E.Ho&Donald B.Rubin,Credible Consul Infamu far Empirical Legal Stud.,z ANN. Rev. L. &Sec. SO q (sou) (discussing ptinaples for nasal inference); john ). Donohue III & Daniel E. Ho, Does Fighting Terrorism Increase Ordinary Crime? A Reexamination and Cautionary Tale(June,,=It)(unpublished manuscript) (on file with author) (illustrating the difficulty of drawing inferences about the effect of increased policing when it was parr of a comprehensive response in a rommist.tock). ss. See BOEHNKE, supra note r8, ar 45; Brox Vdmetri, Res..' Rasing, To Start; Health gymiab RasF.To Post Grade Suckers,ST.PETEmausec Tunes,Sept.a7,r987,at r. S99 THE VALE LAW MURNAL 122:574 znta television exposes of sensationally poor sanitation in several restaurants in November 1997.'54 Although one might argue that this timing is random, a languishing health-inspection system may have facilitated the expose. Even if the imp rus was random, the response was comprehensive. In the months - following the expose,the county Heath Services Department closed restaurants at three times the previous rate in a county-wide crackdown."' On December 9, the Board of Supervisors voted on an array of reforms.116 (In a further complication, each incorporated city in the county still had to individually adopt the posting requirement for grades) Almost simultaneously, the state adopted new requirements for food temperatures.1' The Los Angeles experience shows that multiple, simultaneous policy changes can confound inferences about grading. First, one typical crisis response is to increase the number of health inspectors. Los Angeles immediately hired twenty new inspectors, for example, to implement a "=to tolerance" policy."' Grade reform, relatedly, is often also accompanied by a change in the frequency of inspections. Los Angeles appeared to increase the frequency of inspections, but, perplexingly, restaurants can also pay for an immediate reinspection upon an undesirable grade.19 Second, the reform might also incentivize inspectors to engage in tougher inspections. In Los Angeles, the Department flexed its muscles by shutting down more restaurants, rotating inspectors across the county to prevent any familiarization with regulated patties, and establishing a public hotline to register complaints!"Third, the nature of the disclosure may vary, from posting the iso. Behiod the Kikhm FAar(KCBS vAevision broadcast Nov.16-18,1997). iss. See Hector Tobar&Jeff keds,Restaurant Get a Task of Tough County Hmlth Policy, L.A. Totes,Jan.29,1998,hap://artcles.Ladm .wni/x998[JaD/29/llc s m]-13, . ,56. See County To Tighten Ruta for Rermumw, I.A TrAus, Dec. To 1997, hntp.// articles.ladmu.con/1997/der/IoQod/me6263s isr. See IcemenBeds Rudolph Foodliafay ARof 1997(modified At CAl-FUAUH&SAFETYGORE 414004(West 2012)). mss. Tobar&Leeds,supra some 155;see Meyer,suym note 4t. is9. An owner-initiated inspection is available on¢envy twelve-month period to food-facility owners who want the opportunity to improve their numerical score,letter grade,or both. See LA. COUNTY, C , Core S 8.04339 (zolz); see also Bob Pool, Work Bails Over for Ramumu Lupecroa, LA Turns, July '9, e999, harp.//arddes.ladincscom/1909/)uVt9 /local/me-9458('But restaurant operemrs unhappy with a bad grade can do a fast cleanup and pay a slit fee to apply for a quick reinspection.').The reinspection fm depends on die assessed risk level of rhe ses.t.LA COUM`f,CAL..,CorE S 8.04.768(F) reo. Sce Jonadaan E. Fielding et al., Making the Grade: Clanging the/numioes in Retail Food Establuhmem Fmpecuon, 17 Aral.J. PREVENTIVE Also. 243. 244 (1999) (describing changes with the Los Angeles grading system). boo FUDGING THE NUDGE grade,the score,an inspection summary,or some combination thereof.Fourth, jurisdictions may also simultaneously change the underlying scoring metric, such as California's changes in temperature controls. Fifth, jurisdictions also often increase licensing requirements. In this we, Los Angeles required that managers be certified in food handling."' Crit, the crisis itself may change behavior with respect Eo food safety,both by consumers and restaurateurs.Any health improvement,for example,could be due to the TV expose. In short, grade reform is often confounded with several other policy interventions. For a given jurisdiction,we might more credibly assess the joint effect of the entire reform package, rather than the isolated effect of grading. Jointly assessing the reform, however, also makes it more difficult to assess how different reform packages in other jurisdictions will fare. B.Jurisdictional Variation Despite a federal model food code (or perhaps because of its advisory nature), local jurisdictions in fact administer health inspections in divergent ways. Table r reports results from a survey of[weary large metropolitan areas in the United Scares. (Appendix B lists statutes, regulations, compliance manuals, news reports, and phone interviews used to compile the information in Table r and Table a.) Table r reveals several findings. Inspectors do not necessarily specialize in restaurant inspections. In Los Angeles, some a}o inspectors are jointly responsible for housing, pool, and restaurant inspections, such that the full- time equivalent employees performing restaurant inspections is roughly to6.'m Whether and how a point system is used varies considerably. Nine jurisdictions do not use any formal numerical point system. In seven of the twenty jurisdictions, restaurants must publicly post a grade or some other indicator. Last, the availability of online data on individual inspections varies considerably. In New York, one can view the specific violations cited for each inspection, along with the score and any action taken by the restaurant following the inspection (e.g., issuance of a notice of violation). In Baltimore, on the other hand,consumers can only view a list of restaurants that have been shut down by the health department. 161. See Jill Uovy, Tougher l auuuva Health Coda Urged, LA Tuns, Feb. 12. 1999. http://articles.ladmes.mnVr998/feb/iOb Vmc-ls3zS , . For the more m a prelndatiou of"6dl-time egnivalenr" employes, sec rhe caprion of m Table 1, 6o1 THE VALE LAW JOURNAL 122:574 2012 Table E. FOOD INSPECTION SYSTEMS IN MAJOR METROPOLITAN AREAS XvCCFj �ffJIT•(� F�a9rrc-�S� tj Y „<a. 3 J I Mo0q 110 xao; ✓ i✓ ✓ -1pa6 9 ✓ ✓ ✓ ✓ lO��OAAN CLS ✓ ✓ m6,1 l.6 x401 ✓ ✓ ✓ ✓ ✓ ✓ $+ v IK r ly.aoo ]0 Sy I i nA✓ ✓ r 4 '� 1155001 A 3SI K ✓ �. ✓ ✓ ✓ ✓ vHlte F I ✓ 13609 -,v x6-p ✓ ✓ ✓ ✓ 1x,50 j za 3L 3]I 1✓ * ,✓ ✓ ✓ SP Ny 1fiG0 ✓ IIx. 131M 511 �✓ -✓ ✓ +100<90,�✓ ✓ SEATrt>: ✓ x050 32 '1 11.. ✓ -44 x33I✓ ✓ ✓ ✓ SLA FRANCISCO ✓ 1...JO00' a1 L4: 1. r ✓ +10o ✓ ✓ ✓ MIAMI/OAOE ✓ $600_ 30 x6� j, �✓ J ✓ r aAO.I ✓ SA56 14 ri! ✓ WASXIHG]OM{O:C ✓ 13.Oo0i Ix I]' _. �✓ r F4RA5f) ✓ S,Ooo J9 ✓ +100 <so,✓ ✓ ✓ d}13ilt�i r i 5 j xo 25 r <vo,l r ��xL� 865LUx ✓ A7471 6 O 1J-L8� ;;;.Lf!��✓ ✓ ✓ l I�IfVIFLt ✓ 3.3001 13 41 1✓ 1✓ ✓ J.. 41 ✓ ✓ MIL VKEE ✓ cxa[IOiTE ✓ 3A00, IS 33-: '1 ✓ ✓ 1Sl:LOV15 ✓ ido0l xi✓ ✓ ✓ ao <83'.✓ J ✓ ✓ 602 FUDGING THE NUDGE "Regulatory jurisdiction-indicates at wludl level of gownimend andmriry smuca"MI Inspections of food service establishments(FSEs)are Conducted.I,Las Angeles,bode county and city authorities renin and amity.In both San Fe"MenCo and l,auisvd1s,die city and manly are a unified jurisdiction. 'Esnb mlicrunts" bribe es due number of FSEs inspected, mud may include, depending on lad law, not only conventional resmurants,but also schmal cadi,reds,delis,and grocery stores that serve ready-m-eat food. The "inspecter" Colmeru list (a) the aduni ed fdl-rime equivalent (FTE) of employees devoted to FSE inspections; and (b) the estimated number of individuals with direct responsibility for conducting ou-sire sanitation uupecdors.For-ample,in New York City, although there are All irspecmrs, roughly 78% of inspections are devoted m FSE inspections, making for 1{" FTEs. We report both numbers when inspectors do not appear to specialise -dhsively in food inspections. inspections"indicates die v iuurmn number ofinspectiow that the/urisdiction aims or is redo ned to perform lace yea,, either for all FSEs or die lowest risk FSE when a jurisdiction bases inspection frequency on risk levels. 'nd,lic grading" and "public posting"indicate whether lot law requires due(a)a letter(or equivalent)summary; or (b) any summary of an inspection be publicly posted for all FSEs in the adopting jurisdiction. FA For example, in Los Angela County,all FSEs in a city adopting grading are required in post letter grades. In San Francisco, because only FSEs with acorn inspection histories tie legmred m post a'Symbul of Excellence"it is not dassJed u a grading jndsdiction.The"point system"Column indicates m.dmr a numerical point system is Used to sire irupecuons. Tod points-indicate die maximum some,where a (-) sign indicates that die store counts violations (e.g.,in New York) and a (+)sign indicates that he score represents die degree of complimI-leg.,in San Diego).The -follow-rap tia,ahold"is die.,.be,ofpoillw due trigger a fonow-np inspection.Cells se,gray where not applicable.The mlrmuu for"incams odua"Md.whether (a) all uspecemu (e.g.,Munn and follow-up); (b) individual violations; (c) scores; (d)m,Hom cola,by the hddi audiority in response to the inspection(e.g.,shutdown); or(e)readily dowhdoadable-i—dm are available"dine.These ligina are emblem. based on sources listed in Append"E. The variation becomes even more apparent when examining the design of grading systems. Table 2 reports differences in design elements of seventeen grading jurisdictions.16;While we observe several first-generation graders (e.g., North Carolina, San Diego, and St. Louis), most current n ring ssystem were adopted starting in the late tggos. (In that sense, proponents ropo targeted transparency are correct that such policy efforts emerged in the last twenty years.) Mancopa County,Arizona, instituted grading in goat,apparently disregarding its opinion in xom that "[tjhe grade card does not give the public the complete history of the establishment,but a possible Use sense of security." 163. These seventeen jurisdictions are not an exhaustive list of grading jurisdictions,but simply provide a sense ofthe range ofgnding practices. ,sq. Su6musion for 211r Samurl J. Crumbiur Consumer Froaradon Allard, 1AmucxaEA Couerre ENve.. SEavlees DEET n (2001), http://www.fpi.orgrimaFWtom%xomaricopa application.pdf. 603 THE YALE LAW JOURNAL I22:S74 xo Lx Table x. RESTAURANT GRADING REGIMES Rcr lAt S :�:i 4 (W ton) vno= a mr c..2 ✓ ✓ xy. s o YA gtl9 49 < ✓ <9. 3 uY w G1] SOUT elRNNEVADA l.. 14 •: •• 09 Or . ,ty R. s'41 ✓ >�. s .n ori eg .,. sr g e a sy ✓ Pm- - 3 (sr C 4 -,9y rig sm )9 Eelket eom �,.E Y.lmu9 'I (s-•ul ( 4 �QIIF9 [ 9W i� yr �mr� � Ya^+R^a4 ✓ V„u Sv$la YY "sa•ria si os ntlf[` ', a a la e n c sam r wy. LOU xr+IL�I 1 r t) 9 .. eea9 »n13 383 j 6m1 ' 'fiJpw• ts4 & N. I .. .... A AsW c.a,K r O � .. � u y ex a L <ry ✓ ✓ < s A e e i g ✓ C ,. v 4iFW,yo > ,✓q� n e yN ✓ ,. >a N i (N; 1 ;TA ALRA,NIVCj Hg.. OulxY) • €isRw o- Dtl Fyl 8yL Fu• , m^ v.- ~ o k4. 9J`'y H829 &v <_lkrte a-9 Go4 FUDGING THE NUDGE -Enacrourn is die year (or best estimate) die restaurant grading system was established. "Tend points" indicate die rod points used (with cells gray when inapplicable). "Crit.-and"gen-points-indicate die total number of paints for critical and general violations,with Penin ranges for individual violations in parentheses below (c and g sand for die number of violations).The rte t four cultures Present die posted grade or sign(in rhe color that it appears on the sign—in die online version of flus Article roily),with conrsponding criteria.Wren ajurisdicuoti s terminology differs,we ormalite violations as general or nnoj violations.Maricopa County,for example,uses die terms "prioiny violation"and"priority foondau m violation."Each row of grading criteria is a disjunctive mndn im. For imemo,Marieopa County usigiis a grade of'D'whin there are drece or more critical(c>3)or four or more gcneal violations(g>4)�'Smre post" indicares whether die numerical score is posted in addition m the grade. The reinsPecuori minmes indicate(a)whether rhe first reinspection can change fie grade or some; (b) the impection result or condition that uiggera such a reinspection; (c) days until that mirsspection;and(d)the dollar fee for da reinspection.A"Req.'in die trigger column indicates dot a ce,emo n owner must request a reinspection.The last column indicate the minimum number of bsspectoeis required per year. In Ululate County, grades are posted only online,whereas in all other jurisdictions, grade are Posted on site.The='sign indicates rough approximation. Point scores for types of violations exhibit wide differences.While Georgia and San Diego both employ a mo-point scale,a critical violation can garner up m 9 points in Georgia, compared to 4 points in San Diego. North Carolina awards 2 bonus points for food safety "education credit," a minimum requirement in other jurisdictions. Divergent grade thresholds make it quite difficult to substantively understand the meaning of a grade. In Louisville, a score of 85 leads to an 'A,' while a score of 84 leads to a'C.'The Louisville W cutoff was successively towered over the years: from 93 for 2002 to 2010, to 90 in January zona, down to 85 in September 2ott.i65 Maricopa County allows restaurants'to voluntarily opt into grading,with a dizzying translation of critical and general violations into grades. One critical violation (given the puzzling name "priority violation") and two general violations ("priority foundation violations") result in a 'B';"' zero critical violations and three general violations result in a 'C.' Latimer County, Colorado, uses—in addition to the state's loo-point scale—a 569-point scale for grading purposes. The county oddly explains that the 569_ -point scale is "based upon similar models used in r65. Ser Gil,supra note 40 (noting in soma that"Ir)estaucants will can an A for a score of too- ,;"); press Release,Louisville Metro Dept of Pub.Health&Welbuss,Public Health and Wellness To Resift Food Placard System, Aug. z5, sou, hap://wwtv.lonisvdleky.gov /HealdVNcws/hr-08-plarards.han (announcing a revised 'A' cutoff of 85): E-mail Corcespondence with Gretchen Boyd, Envd. Health Supervisor, Louisville Dep't of Pub. Health&Welhoss(Mar.a3,xoha)(on file with sudor)(noting the change of the'A'mtoff from"m 90 iia sou). 166. The new terminology come from the long model food code.Sm U.S.Put.Heu.TH SERv., Poor&DRUGADMIN.,'Vra note ma,at x-xi. 6o5 THE YALE LAW JOURNAL Iss:S74 30 13 other parts of the U.S.ni67 The nature of the disclosure also differs considerably. Albuquerque uses a green/red sticker system.Toronto uses a color-coded sticker system: green for a passing grade, yellow for significant infractions, and red for major infections.Maricopa County has a rate letter grade DPW Across jurisdictions, a'B'grade may be posted in the colors blue,green,yellow,or black. (Denmark, not in Table z, uses smiley faces.i6') All jurisdictions where grading is mandatory establish some soft of reinspection system for grade adjustments. C. OurApproach As Table z underscores, the impact of grading is unlikely to be homogeneous across jurisdictions. The nature of health inspections, scoring, and disclosures differs in such sharp ways that the effects documented in Los Angeles are unlikely to hold for otherjurisdictions. This Articles approach is to examine previously unstudied jurisdictions to assess the efficacy of targeted transparency beyond the extant case study.We focus primarily on New York and San Diego. Together with Los Angeles, these comprise the three largest grading jurisdictions in the United States (see Table 2). The size of each of these jurisdictions also means that we have rich microdata from a large set of restaurants and inspections with which to examine the implementation of grading. New York has the particular advantage of having instituted grading during our observation period (in July zom), thereby also providing us with temporal variation in grading. San Diego, on the other hand, permits Ls m examine the practice of grading in a system that has existed for many deiades.i69 Although we sought at the outset to design an evaluation of grading's Effects on sanitation and health outcomes, our analysis of the data revealed far more fundamental conrems with each of the systems. Do grades convey any 167. Ronna zea Inspection Domhatt, IaNMek COUNTY HEALTH & ENVT, hap:// www.m.larimer.m.us/food/aboucasp (last visited Jan. 30, 2012) ("In addition on die traditional Too point scoring syson which has been used in the past, LCDHEd3 Jus developed in own food establishment rating sysmn called the 'Risk Index.' This rating system is based upon similar modds used in other parts of die U.S. ... .Like golf scores, the(ower the risk value, de be.the sanitation level . . . .There arc 569 possible value points in du risk index rating system, 530 are for critical items and 39 for non-cronal i¢ms."). 162. S4 Irmo fo S-ikY SyAon,DANISH VensamasY&FWD ADMIN.,hup://www.find5mdey.dk /en-US/Forsideturn(last visited Apr.l6,aou). .6s. Of manse, what we find in these jurisdictions may not be represenutive of restaurant grading elsewhere. 6o6 FUDGING THE NUDGE information at all?Are inspection scores meaningful?What effect does grading have on the regulation of food safety?Our findings suggest that proponents of targeted transparency have drastically overestimated the salutary effects of grading and failed to notice its costs. While compiling and structuring the dam (e.g., with the complex classification algorithm we developed for New York inspections) required considerable work, we relegate these details to the Appendix. More sophisticated statistical methods, which we propose in Section VILA, may be applied, but we focus on the core findings below,which can be communicated with minimal mathematical background. We turn first to San Diego, as the findings are simpler and provide an important point of comparison for our findings in New York Appendix D demonstrates that our results generalize to eight other jurisdictions. 111. FUDGING BY INFLATION: SAN DIEGO We begin by sketching some brief background on San Diego's health inspection system in Section LILA. Section IILB describes the inspections data for San Diego restaurants,and Section HI.0 presents results. A. Regu aazny Background California's Retail Food Code sets statewide standards for food safety in restaurants. The state code establishes substantive violations, following the soon FDA model food code."For example,the code specifies poultry"shall be heated to a minimum internal temperature of t65 F for IS seconds.""' primary enforcement responsibility falls upon local California agencies, in San Diego's case the Food and Housing Division of the County Department of Environmental Health.The agency is self-funded by permit fees,with a budget of around$6 m0lion;'x and is responsible for administering the food code for roughly Is,000 remil food facilities,"' including not only conventional no. S.Ma'nofFood&Ding Officials.Red Fmgov in Food Code Adopunn,U.S.FOOD&DRUG ou h fries dovmloadsIN- /FcodSakc//RenilFoodRoaction /pe&+ Qoly,t,x ), up�//s*"s'v'- CM- 6.hun. /FederalStazeCooperauveriograms/UChfx3o33 m. Ca..HENTH&SeFMCooES 1w0o4(a)(3)(Westxou). ,r.. Gary W.Erbedc,Samuel J.Cmmbim ComumerFr tiim Auwd h[5 SuumiVml,Court OF StN DIEGO Da,'T of ENva. HEN.TH 3 ( o 5). 'P:// fpt.ocglm g s /san%2odiego%xoapplkadon.pdf. s. Sm Food Facility Imparion Search, CouN of SeN D1EGo, hnp.//w .sdcounty.ea.gov /dch/fhd/ffisftnrro.html(lut visited Jan.31.201x). 607 THE VALE LAW JOURNAL 122:574 2012 restaurants, but also any establishment Serving ready-to-eat food (e.g., convenience scores, school cafeterias, and health care facilities).'''The primary inspection vehicle is a "routine inspection," which is comprehensive and, as in Los Angeles, scored from o to too points, with Loo points indicating no violations (full compliance). Common violations include failure to abide by "holding" temperatures, dirty food-contact surfaces, improper cooling, and failure to wash hands.Violations are classified as major or minor violations, the former receiving higher point deductions. For example, failure to heat poultry to 165 F is a major violation resulting in subtraction Of 4 points, while failure to wear a hair restraint is a minor violation resulting in subtaoion of 1 point.171 Limited (or "directed") inspections are unscored.j6 These are typically reinspection after a notice of violation was issued to rhe facility or inspections of establishments that have limited food service areas(e.g.,grocery stores). San Diego has graded a subset of food facilities since 1947.+"Under county code, "[a]fl restaurants, bars, taverns, retail food processors, and deli markets where food is prepared will receive a grade card.'"'' (Establishments that are scored and inspected, but not graded, include grocery stores, gas stations, liquor stores,and schools.)The grading system assigns letter grades of'A,"B; or'C,'if a routine inspection resulted in 90-too, 80-89, or 79 or fewer paints, respectively.'" A 'C' is considered a failing grade and may result in penalties and permit suspension if the restaurant fails to achieve a some of at least 8c, points within thiry days."(The Department of Environmental Health and the Housing Division can always shut down a restaurant if it poses too large a public health risk.) The county employs around fifty inspectors ("Registered Environmental Health Specialists"),who conduct housing and food inspections,with roughly 75-80% of time spent on the latter.' ' Inspectors are required to (T) have a +TI CfiL HF. T`H&SAFETYCODE§n3789(West.JJn. Sm Road Fmd Facility Opermor's Gndde,mpmno¢n3,at6. mi. See Food F"Lity Inyeetlan Search:Direuul, COUNTY Or SAN DisGo,http://w adcolm y .n.gov/deft7Flld/tfis/insp_resule/dirated.hmll(laze visited Ap,z,,con). Ln. Su Williams&Almendaria,supra note 88. vs. See Retail Food Facility Operator's Code,supra note 113,at xq. p9. SAN DranomvrY• C ,CODE OF MG.ORDINANCEC§6r.107(a)(2012). "o. Id.§6a.1O7(b)-(c);Retail Food I lity Ope awr's Guide,mar.nore Lg,at x8;Food Program, CouN Or SAN Diuca,ltttp.//www.sdmuncy.ca.gov/dd,/fooQffood.ht.1(last visind Jan. 31,201x). 191. See Williams & Armendariz, mpra now 88; E-mail COnespondence with Bao Huyuh, Supervising Envd. Health Specialist, Food&Hous. Div.,San Diego Cory. Dep't of Eovd. Healdt(Apr, u,2012) (on file with mlthor); Food Facility Ivpection Searck,supra now 173; 608 FUDGING THE NUDGE baccalaureate degree, with thirty semester units of basic biological, chemical, physical, or environmental science; (2) have "[ojne . . . year of experience investigating and enforcing environmental, public health or sanitary laws and regulations" or "[sjix . . . months as a[j . . . Trainee with the County"; and (g) pass a state civil service exam. Starting salaries range from $50,000 to $61,000."' As a general matter, the county's food safety system is reputed to work well. In zoos, the Department received an award for food protection.i" With respect to transparency and the grading system, however, performance is less clear. In 20o6 to 2007, the county convened a grand jury investigation to examine why a seemingly large number of restaurants were receiving'A's.The investigation did not examine large-scale data, but conducted a series of interviews with inspectors, health officials, and restaurateurs. Noting the ratio of then forty-eight inspectors to 12,000 establishments, the grand jury concluded that "[pjressure is high for . . . inspectors to manage regular visitsii8' and recommended increased funding for hiring inspectors."' It also found "no formal means of communication for informing the public of restaurant closures and the outcome of subsequent impectionsniBB and recommended the creation of a website of inspection dam"D San Diego Crary. Grand Jury 2oo6-2007, 'A" Grades in San Diego Cousdeo�mumau Deserved m Not?, GOUNTv OF Sn DIEGO 3 (May 7. 20n7). up:// ry.cagov /grmidjury/reports/2oo6_2oo7/msraurantgrade.pdf. sea. See Erbedr,mpm note 1R,.16;job Desmplio.6 Solaria,Enuimnmenral Health Specialist 1, Country or Sau Duno, http://agency.govemmengobs.waVsdmnnry/defatdecfm?anion =viewdunper&duSSpecID=807 °^ucy=s4o8S& imordy-yes(Ian visited Feb.3.201x). Samadj.Crum6ins Cammmen,ProactiouAteard:Parr CrumbintAuwrd Winner,FooDsenvoe PnexeGwG INSL,http://wwsmfpi.orgflmages/past%zocnrmbine%zoaward%zowil s.pdf (last visited Jan.31,201x).But see PE&COGx,snort nom it,at 344 (amigmng San Diego a grade of C'based on a qualitative survey). 184. San Diego Crory.Grand Jury 2006-1007,supra nom ISI,at 3. ins. Id.a14. 1s. M.at 3. is, Id. at 4.the grand jury focused proudly on ustanrznt tI.,um, by die depammenq as is evident from the types of information the grand jury concluded the websim should contain (i.e.,rescaurant come mid location,daceof closure,reason for closure,date mopened,and result of follow-up inspection). As vee ague in Part vD, on iuspemon data should be disclosed. 6oq THE YALE LAW JOURNAL 122:..4 2012 B. Inspections Data To empirically study the San Diego grading system, we downloaded and extracted into proper database format all available inspection reports from the Department's website at the end of November 2011.'2'The data include 37,040 inspections of 11,941 establishments between June zoo? and November eon. Information includes the date, type of inspection, and score (if applicable). - Figure z plots the distribution of types of inspections. Just under 80% of inspections are routine inspections; just under 2o% are directed (unscored) inspections; and a small number of inspections are other kinds of inspections (e.g.,reinspection resulting in the restaurant closing or reopening). Figure 2. INSPECTION TYPES IN SAN DIEGO P.eopcn Moore Dircc ed(initial) Dircerm(miuspecrion) _ Routine 0 5.000 15,000 05,00. Nmnlber Routine itupettions arc scored and graded. Directed inspections are limited in snipe and unscored and ungraded.Some facilities(e.g.,grocery stores)may receive a routine inspection of areas where ready-[ocat food is prepared and directed impectiom in du remainder of die facility,such that routine and directed inspections occur during the same inspection visiL Closures may occur across di&cent types of inspections. Reopening inspections occur only for resnurams du[have hcem closed. ou. See Sam Digo Food Facility rnpAu.. Smash, C LINTY of SAN DIEW, http://wwwssdcoanu.m.gm,/llss(last visi.d Ism 31,0012). 6m FUDGING THE NUDGE C.Empirical Results 1. Uniform Grade Inflation Figure 3. INSPECTION SCORES IN SAN DIEGO °o � a IL O O o Bo 85 95 Io0 Scot Tlus lilsmgran depicrs sorra for 11,813 Sm Diego msmnrmrs-of November zou. we first examine how the grading system operates in the most recent inspection cycle for restaurants. Figure 3 displays a histogram of the most recent inspection scores for San Diego restaurants from November zou. The black vertical line at 90 points shows the threshold for receiving an 'A.' Each bat represents the number of restaurants receiving each raw score. The mean score is 96 (standard deviation=3),but there is a sharp discontinuity at the'A' threshold. While 703 restaurants received a score of 90, only two received a score of 89.Out of 8,941 graded restaurants,only eight received a'B;and none received a'C.n° If the theory of targeted transparency is to provide a signal to consumers to differentiate amongst goods,San Diego's system appears to fall short. Because 99.9% of restaurants receive an 'A,' there is little variation from a consumer's perspective in the sanitation level of restaurants- ,ev. OnroCu,8r3 smred inspeaimu,ody fi@un esmblishmena received a score Ixlow 90. 611 THE YALE LAW IOURNAL 122:514 a012 That said, the high proportion of'A's alone is consistent with both benign and nonbenign behavior.It may be that San Diego restaurants have responded to the grading system by improving cleanliness precisely to target the threshold of 90 points. But consider two other points. First, county law permits restaurants to pay a fee ($142 per hour) to be reinspected, and possibly regraded, within thirty days."* Strategic cleanups for regrading arguably are much less likely to yield general health benefits. Out of sixty-two graded restaurants that have received a score below 9o, the median time to scoring above go points is seven days. Nearly a quarter of restaurants achieve an 'A' within one day of the original inspection, and 8o% do so within a month. At any given point of time, the probability of observing a'B'grade even for these establishments is hence extraordinarily low."' Second,as the public health literature emphasizes, inspector discretion may well explain the discontinuity. Although scoring is conducted via a specific worksheet, there is discretion in which violation to cite and, in many instances, how many points to assign the violation. How would one determine, for example, whether"[flood contact surfaces [are] clean [and] sanitized" and, if not, whether it warrants a two- or four-point deduction?" Health officials report that converting scares to grades induces a form of"ethical stress[]": "operators are likely to demand that their political representatives or the courts intervene, or they may be tempted to 'encourage' inspectors toward good grades through unfortunate and unethical means."193 Most tellingly, one San Diego inspector revealed: "Some inspectors will give out a B for an 89 . . . . I usually warn somebody at that point.It's ajudgment call... ."'94 We explore the role of inspector discretion more formally below. X90. SAN DIEGO COUNTY, GL, ConE of REG. ORDINeEtus g 61.I01(b) (2012); Retail Fond Facility Operators Guide, supra now 113. at 29; Dept of Envd. Health, Food Facility Fee Sohdule Ef doe August a4, aou, Couory OF SAN D cGo, hnp://www.sdmunry.agov /dehffomi/pdf/publicadons_fcewhedule.pdf (last visited Sept. a, 2012); Telephone Interview with Celia Kroy,Envd.Health Specialist&Specialist on Duty,San Diego Dept of Envd.Health(Jan.19,2012). 191. The same pattem holds for closures of restaurants. Out of sixty-seven graded raramants that have been closed,die median time until dere restaurant was reopened was die same day. Four out of five graded restaurants that are closed reopen widen one day.Now that such regrading alone does not account for the discontinuity at 9o.The difference in the number of resrzumnts just above and below the threshold of 90 points is much larger den the number ofwaraurs nrs paying for reinspection. 192. Sea Rawl Find Facility Opento is Gsdde,supra more 113,at 6. 1R3 BOEFINKE,rupra go.)S,ar 4-x5. 194. Ed Sylvester,Making Sure Your Eating Plates Are A-OK:Impecion Rare S.D.Reman-ams,LA. Thus,May a5,1980,a1 Ah. 611 FUDGING THE NUDGE Regardless of whether restaurants are precisely targeting the threshold, the practice of grade inflation in San Diego means that its grades fail to convey to consumers any substantial heterogeneity in sanitation. Uniform'A's at the very least appear inconsistent with the Departments justification for grading. The Department states, "local consumers and visitors quickly learn the usefulness of the grade in selecting a place to dine.""' Like Lake Wobegon's children, San Diego's restaurants are all"above average." 2. Scoring Consistency Another basic way to assess San Diego's inspection system is to compare restaurant scores over repeated routine inspections. If systematic differences in sanitation between restaurants exist, we would expect such inspection results to exhibit substantive correlation over repeat inspections.To be sure,we would not expect (or desire) complete consistency: after all, a low grade should incentivize a restaurant to dean up, and a high grade might cause a restaurant to be less vigilant. In addition, chance factors, such as when the inspector shows up and what food is being prepared,would attenuate the correlation."" But if grades based on the most recent inspection report are to have any meaning, they should minimally convey some substantive information about the prospective cleanliness of the establishment. Figure 4 plots inspection scores from routine inspections for the same restaurant across the first two observed inspection cycles."' Each dot represents one restaurant, with the score it received in the first cycle on the x- axis and its score in the subsequent cycle on the y-axis, randomly jittered(and transparent) for visibility. The top panel plots these for the full range of observed scores from 61-too points. Because restaurants that score below go points have an incentive to improve sanitation practices and are thus likely to reduce the overall correlation, the bottom panel excludes these and focuses on the W range of restaurants. if sanitation measures were perfectly correlated, ys. Erbeak,n pits note rR,at Is(emphasis added). hgs. In statistic mars,we of wane Expect some regression in die mean,but die correlation weffcimt provides one sense of the relative weight of systemadc—restaurant-specific— factors relative m smchasdc faamrs. ,qr. To be precise,Figure 4 plots scores across the first two observed repeat rutin inspections for swrcd(but not necessarily graded)establishmears. Results are substandeely die same for the subset of scared and graded restaurants.There arc slightly more restaurants scoring below go than in Figarc},which plots the most recent nispecnon score as of November 2011,while Figure 4 plan the first two matin,mspectioa fault 2007 onward Botanic the frequency of inspections is tied in the risk of the establishment,we do not plot all purrs of impemmu,which would over-represent high-risk establishments. 613 THE YALE LAW ICURNAL 122:574 zotz dots should line up on the forry-five-degree line, and the proportion of variation explained by prior scores (R') would equal one. We find that inspection scores exhibit substantial consistency over time. Restaurants that received a high score one year tend to receive a similar score the subsequent year; there ate very few data points in the Lop-left and bottom-right quadrants representing restaurants flip-flopping scores across years. A one-point increase in cycle r is associated with a half-point increase (plus or minus 0.02 at a 95% confidence level) in Cycle 2.'9a Roughly a quarter of the score variation in the second cycle is explained by score in the first cycle (R'= o.z4 for the full range and o.27 for the 'A' range)."' This level of consistency appears roughly comparable to that of Los Angeles restaurants.— Figure 5 displays more substantively the information in San Diego's scores. The left panel plots the distribution of stores in the second cycle for restaurants receiving below 95 points in cycle I, while the right panel plots the distribution of stores in the second cycle for restaurants receiving 95 or above in cycle I. The distributions sharply diverge. If a restaurant scored below 95 in cycle I, it has a 47%chance of scoring a 95 or above on the next cycle, compared to an 8o% chance for a restaurant scoring above 95 in cycle z.-'In short,a restaurant's sanitation score is informative. 19s. Tlus is based on a simple hour lease squares regression of scores in le z as die outcome P 9 S y and some in cycle r as the main explanatory variable for the full dataset. 199. Sm supra now r98.The same model is fit in the subser of'A'-graded mortuaries. aoa.S supra were accompanying nate 14;(finding in an earlier study by Jin and I lie that just under a quarter of the variation in tcsmurant impections is explained by time-invariam wstauram-specific attributes). vat. It is possible that anchoring bias explains die eausismncy in San Diego results if inspectors arc aware of the previous inspection score. but the real puzzle,as vrc explore below,is the relative consistency in San Diego and relative ii sistency in New York In bodr jurisdictions,inspectors appear m have access m prior inspection results. 614 FUDGING THE NUDGE Figure q. INSPECTION CONSISTENCY IN SAN DIEGO FULL MNGE �W R'= o.aq 8 60 to ea 9" hoo Cycle r X 6ANGF is -e5M 0 V�f«tx9 fA M i. qo 9a 94 96 98 roo Cycl<s Each dot represents one«,...rang wirh the snore it ei<ived on its firs'rondne inspection on die s-axis and the score from is nut routine inspection on die y-axis.For visibility, observations are rantbmlyinnied.The top panel depica die fill range ofobserved snores and die botmm Panel depicts restaurvta receiving A'grades in both cycles.If measures were perfidy correlated,doa vmuld line up on the forty-five-degree line and A'would equal one. Roughly a quarter of the vaiiaton in irapecoon scores is predicted by prior iosprttion snores (see R'in he lower right hand corner),so inspouens are rmssunng some degree ofsysrematic sanitation difference amongst retanrana. 6LS THE YALE LAW JOURNAL 122:574 lora Figure S. PREDICTIVE POWER OF SCORES IN SAN DIEGO CYCLE I SCONE<95 CYCLE 1 SCORE 95 C O� O O� C; T o T 90 95 100 90 9^5..1 loo Soore in eydI 2 Sr.,0 in ryde a The left panel depict rhe..dambudon to ryde a,given that a resnurant scored below a 95 in tyle t.The right panel depicrs the store distribution in ryde a,given that a mountain stored 95 or above in ryde 1. The above results paint a mixed portrait of San Diego's grading system.It is possible that grading over the course of the last sixty years—and the Concordant stability in Compliance expectations—might have caused San Diego restaurants to improve sharply to 'A'-levels. San Diego might then be a resounding success for targeted transparency. The sharp discontinuity at 90 and the ability to request a regrade within a day, however, call this interpretation into question. While San Diego health inspections exhibit some degree of Consistency over time,without any substantive grade variation to speak of, consumers—at least currently—cannot rely on such grades to inform their restaurant choices. To understand these results better,we turn to New York. IV. FUDGING BY NOISE: NEW YORK We begin, again, with some regulatory background about New York's restaurant-inspection system in Section IV.A, focusing particularly on the system as it existed immediately before and after grading was instituted in July 2010. Section N.B discusses the inspections data, with Appendices A and B detailing data integrity issues and the classification algorithm we developed to address the fact that New York fails to disclose types of inspections. Section IV.0 presents results. (Appendix C shows that findings are robust to scoring changes, different types of inspections, the time period or inspection cycles examined,and the role of administrative hearings.) 6r6 FUDGING THE NUDGE A.Regulatory Background I. /nspectiONS zoos TO 2010 The Bureau of Food Safety and Community Sanitation (BFSCS) in the City Department of Health and Mental Hygiene (DOHMH) administers the city's food safety program.—The Bureau inspects thirty-two different types of facilities (e.g., senior centers, public schools, correctional institutions, apartment window guards"'),but roughly three-quarters of its inspections are Of restaurants.'°" Its 2007 budget was $11.7 million,"' and its staff consists of roughly i8o full-time positions."' The frontline employees Fire the health inspectors ("Public Health Sanitarians")."a Qualifications are comparable to San Diego's. Inspectors must pass a civil service exam and possess either baccalaureate degrees with at least thirty semester credits in biological or physical sciences, or associate degrees With twelve semester credits in biological or physical sciences and five years of experience as public health technicians. Starting salaries range roughly from $40,000 to$$0,000. 00 aa. See road Safety and Corrrmuniry Sanitarian, N.Y.C.DFP'T OF HEALTH&Mw HYGIENE, bap://www.nyc.gov/h—V&h/h.Minsp.,/insp.shcml(Iastvisited Feb.r, rom). 203. Window guards are child-safety guards on alcomicat windows. aaa-See BUREAU OF MONT.AUDIT,OFFICE OF THE COMPIROaUaA1 CITY OF N.Y.,AUDIT REPORT ON THE DEPPRTMENT OF HE tM&MENTAL HYGIENE OVERSIGHT OF THE CORRECMnON OF HFALTH CODE VIOa VONs AT RFstnuMNts r (mog) (hereinafter aoog AUDIT]; Office Of Community Sanitation: Spacial Population Imprrnon Program, N.Y.C. DEPT OF HEALTH& MENTAI.HYGIFaaE,http.//avwsv.nyc.g.Vhmd/dol,/htmVkmpect/wmm-san.shrml#spip(lot visited Feb.1,ion). 105 See aoog Executive Budget, N.Y.C. DEFT OF HEALTH & Wwat HYGIENE y (2009), hap://wuncu.w/c.gov/hm Vinidadvc/FYo9PBB_MayOpdaWo5rao8_FYo9_DOHMH.pdf. rob.Sm Gleam Collins, Restaurant Grading Begim in Nov York, N.Y. noses, July a7, 2010, http://w ,,YdmeS.MnVZOIO/07/28/nyregiOLV/ iisspect.ii.l. Wlvo New York's Indepaadenr Budget Office(IBO)reviewed Ciry Depmtmaa of Heald and Menta Hygiene (DOHMH)'s budget programmatically,it actually fisted die budget for die Bureau of Food Safetyand Community Sanitation (BFSCS) as $34,,000 with one full-time position. Instead, MO allocated sanitarians in "General Eavuonmenral Heald" See IBO4 Programmatic Review of the aoob Budget m of the November Financial Plan, N.Y.C. INDEP. BUDGET OFFICE 21 (2006), hup.//www.ibD.nycny.us/iboreports/agem]'Budgewo5 /D0HMH%20PMgmm%2oBudget.pdi Telephone Intervicsv wilt Doug Tumulty,Chief Of Staff&Cmi mens Dir.,N.Y.C.Indep.Badger Office(Jan.10,ae02). 207. See 20c,AUDU7,Supra note 204,at 3. rer, Notice of $nunnation: Public HmUh Sanitarian, N.Y.C. DEPT OF CorvionE ADMIN. SERVICES APPLIGTION UNIT (ama), hrrp://www.nyc.gov/hrniVdcu/downtoadVpdf/mcs 617 THE YALE LAW JOURNAL 122:$]4 a01a As in San Diego, substantive violations in New York City largely derive from the state sanitary code, with local officials bearing primary responsibility for enfotcement�99 In contrast to San Diego's longstanding grading system, however, New York's method of scoring inspections is relatively recent. New York first began to use a numerical scoring system (without grades) in March 2003,"°the procedures for which were formalized in a rulemaking in February 20o,5."' The point scoring system, which has remained largely the same since zoos, aimed in part to "provide a more objective method of evaluating . . . public health risks.""' The point scoring system works as follows. Violations are classified as either "critical" or "general" violations. Critical violations are those "more likely . . . to contribute to food commination, illness, or environmental degradation."�° As of 2005, sixty critical violations fell into one of six substantive categories: administration, food temperature, food source, food protection, facility design, and personal hygiene. Thirty-eight general violations fell into one of seven substantive categories:vermin or garbage,food source, facility maintenance, documentation, and three tobacco-related categories.Each violation could range in severity("condition")from I to V;the conditions determined point scores, which ranged from two to twenty-eight points per violation. In an initial inspection—a full, unannounced sanitary inspection conducted roughly once a year—inspectors chose whether to cite a particular violation and determined the severity (or condition) of the violation. Table 3 provides examples of common violations.For example,a 2B violation for failure to hold a hot food item at or above 140 F could be assigned seven to twenty-eight points, depending primarily on the number of food items s0 held. "Two hot food items out of temperature," such as "8 chicken wings and Looked rice," /2ouoio12000.pdf; Public Health Saulmrian Serie, N.Y. ST. D&T OF HEM.TR, hmp://www.heOtILny.gov/prcvenomypublic_heald._works/steers/public_healrh_saniranan .h,m(las,visited Feb.1,nma). a29N.Y.COMP.CODES E-&BEGS.tit-10,5 J4-1(199a). a,o. See N.Y.Smte R.I.Ass'.Y.N.Y.C. Dep't of Health&Menal Hygiene,798 N.Y.S.ad 711 � (N.Y. Sup. Ce 2004); see also Jennifer Steinhauer, New Rermno n, Rake 1 wlated City Charter, A Judge Detain, N.Y.TiMFs, Ocr. a9, sun, hrtp://www.nydmes.mm/aoo4/10/a9 /ayregion/agresraunnchurd (describing judicial decision requiring public twe nen,before the adoption ofscoring system). an. See Comm'r of Heal,h&Mend Hygic.,Moos of Adoption of an Amendment w Tie,a4 of the Rules of the City of Nna York Adding Chapter 23 ("Food Service Fsmblishment Inspection Pmcedares),Us CRY REG 883(Feb.18,aoog). an. Id. "a. Id.at 884. 618 FUDGING THE NUDGE would constitute a 2B Condition II violation with eight points, while "[t]hree hot food items out of temperature," such as"8 chicken wings, cooked rice and roast beef," would constitute a 2B Condition III violation with nine points."4 Failure to correct any public health hazard during the course of the inspection would result in an automatic twenty-eight points. 's When an inspection resulted in one or more critical violations or fourteen or more points, inspectors issued a notice of violatioa.ii6 Each violation cited therein would carry a penalty of between $zoo and $2,000."' An inspection resulting in twenty-eight or more points was considered a failed inspection, triggering, "whenever practicable," a "compliance inspection.""' DOHMH policy was to conduct compliance inspections fifteen to forty-five days after the failed initial inspection.'9 Failure to remedy violations after two such compliance inspections would increase the chance of DOHMH shutting down the restaurant, although in principle a restaurant could be shut down at any time for posing a public health hazard. The pre-2010 inspection system contained one additional carrot and stick The carrot was a"Golden Apple" award issued to any restaurant that, among other requirements, "passed two consecutive annual inspections with no critical violations and fewer than four general violations."" The stick was an "Accelerated Inspection Program," which increased the frequency of initial inspections for restaurants that had failed two consecutive initial inspections."' Between zoos and 2mo, DOHMH made only relatively minor scoring Impeaion Smn g System foe F sd Sov( Establishuuu,, N.Y.C. DEPT or HEALTH & MEHrnt HYGMW zr (2005), hrrp://www.nycgov/Inmt/doh/doemloadS/pdf/u pect/'nspm -Food-safely-bookpdE: "s. Id.atq. n6. ria CITYP EG.884(Feb.r8,2o01). "r. N.Y.,N.Y.14MTH CODE 5 311(2012). "e. 132 Cm REc.884(Feb.18,x005). 2zs. See xooq AUDIT,supra MM 204,U 9. azo. Pius Pderse, N.Y.C. Dep'r of Heal di&Mend Hygiene, Heal di Department Announces "Golden Apple" Award for Rutauranrs with Superior Food Safety (May 6, 2004), htrp://www.nyc.go/luml/dob/htH/press_uchiveo4/Pro4&oso6.sheul; w zo09 AMIT, supra note xo4,at 16-p.Gulden Apples fell our of use after letter grading was introduced in roto.Compare Impeeriou Sruring Syaem Jet Food Servve E.bhshmew,supra nate 214,a1 8 (describing rhe Golden Apple initiative), -uh VJh , Te Experr FAm You re Insp,,W: A Guvle for Food Suvie, CiPerstan, N.Y.C. DEPT or HEALTH At Mervrtt HYGIENE 8 (amo), http://www.nyc.gov/html/doh/dowdoads/pdf/rii/blue-book.pdf (never menuoNng tlrc Golden Apple). z2,. 2oo9 AUDIT,supra note 204,at 5 n.x. 61g THE YALE LAW JOURNAL 122:f]4 1011 changes to the inspection system."'The only notable change, implemented in July 2oo9, was that DOHMH stopped scoring administrative and documentation violations."' Violation to in Table 3, for example, would still be cited,but no longer scored. an. In Demmber zoos,DOHMH clarified that violations of New Yorlps ban on artificial vans- fat foods would not count for saturation swung purposes,although inspectors would um establishments for tech violadons.See Notice of Adoption A Amendmew to Chapra z3(Frond Service F.rtablirhss nt Inq ation Procedures)of Title 2,1 of the Rules ofthe Gry of New York,134 CITY Rec.5,085(Dec. 27, zool). New York similarly introduced done posing and food allergy disclosures for certain restaurants in 20og and 2oo9, respectively. Sea Comm'r of Merril Health&Hygiene,Notice F46ptior ofA Nov Chapter V(Food Allergy Information) in Tule 24 ofrlre Ruler ofthe City ofNno York,DEPT OF HsALTH&MENrAi.HYGIENE(snag), hega://www.nye.gov/hmd/dob/dovniloads/pelf/notice/food-allergy-inforntadon.pdf; Press Release, N.Y.C. Dept of Health &Mend Hygiene, Board of Health Votes To Require Chain Restaurants To Display Calorie Information in New York Gry (jan. u, song), hrep://www.nyegov/Ihtml/dolh/heml/przooe/prong-oeshntd. In Oerober song, to harmonize its inspection system with state amendments,DOHMH added two violadons for "redu¢d oxygen packaging" (a method of removing oxygen when a oring food w limit Clostridium bomhn nn)and four violations in facilitate HACCP management,and amended in small pav the language of Four cxiadng violations. S. Comm'r of Hddh & Mend Hygiene,Notice eFAdopeion ofAmendonor ro Chapter z3(Food Service Establishment Sanitary Inspession Prendurcs)IfTirk 24 of rhe Rules ofd.City of New York,131 CITY Ree.3.19(Sept. 29.x009). aa3. See Notice of Adaption ofAmend�ticus to Chapter 23(Food Service Esrablishmaa Procedures)of Tule 24 ofthe Rules ofthe Ciry ofNno York,136 CITY Rea 2,64S(jnue 23.2009). 620 FUDGING THE NUDGE Table 3. SAMPLE VIOLATIONS IN NEW YORK(2005) TYPEOFVIOLATION CONDITIONS CRITICAL VIOLATIONS(60) It III IV VI ro N Cur¢m nGd Pcrmi rcgismcion ar orb<r - - - - z8 aorhori zrion m oples a nm a allzble. xB Hor(oodimen mn h ldazo abovey9°F. y 8 9 1. 4 3D Canned food pmdua olwreed swollen,leaking, ] 8 9 to a8 d ruxml. 41 Food inan spoiled, shtherned.wnwrvmred,or A T 8 9 ze 5C Food-mnrazr evrfa<e impooldy consunad or ] 8 9 to z8 I9cared Unsecapetble maze ulused- ONfr'Fob ,1!i9em3^rl' ea ., : "-.aE1♦d l -L.> S.,I CerloralTtered,dr 6A Persowldmnlinas inade . wom Clean tenser 5 6 I 8 - garmcnn.e0auvc hair renszraimror . 8A Fedlirynorvcrminpr f.Harborzg<ormndidons z 3 4 5 on ociero wnun mist 9D Food ton.sudxe nor poopedy maindned yyx 3 4 5 . .,....,..3141tl3 CoA ToilecfariGrynormainrvncdmdpmvidrdwirh z 3 4 5 miler papeq wua rarpazle,and self-dosing door uA Permit noe reotprenerely displayed. - - - "Condidmas"indicans the sevedry of the violation and asseNamd point valim.In meal, there are siery ai6c2I violations and thirty-eight general violations. 6v THE YALE LAW 10URNAt [22:574 2012 z. The aoog Comptroller Audit In 2009, the City Comptroller conducted a performance audit of the restaurant-inspection system for the 20o8 fiscal year."' The Comptroller interviewed officials and reviewed samples of fifty restaurants that were not inspected during the year, sixty-two restaurants that failed one regular inspection, thirty-nine restaurants that failed three or more consecutive inspections, and twenty-one restaurants in the Accelerated Inspections Program.''° The Comptroller made numerous findings critical of the system, but, most relevant for our purpose, concluded that "DOHMH did not adequately trach its inspectors or supervisors to ensure that inspections were being properly conducted and monitored.ni°Examining the internal DOHMH inspections database, the Comptroller found major limitations to the database. For example, "DOHMH officials provided a list of 194 inspectors," but the "database file identified 280 inspector codes,""7 an unexplained excess of eighty-six inspectors. DOHMH's cryptic explanation for the deviation was the presence of "input errors.""a More shocking, given that inspectors are ostensibly assigned randomly, was the variation across sixty-seven inspectors who conducted more than one hundred inspections in the year. The average inspection score was 25 points, but the audit uncovered some inspectors with average scores of 15 and others with average scores of 5o."' 3.Letter Grading In July nolo, as part of the Bloomberg Administration's push for government transparency and digital modernization,DOHMH instituted letter grading. (Mayor Bloomberg's slogan: "In God we trust. Everyone else, bring data.""')The primary change was to convert each inspection score into a letter aaa. Sea20og Atmn,supranomxa4,at1 any Id.at 4-5 aas. Id.at 1. 227. Id.at 14. ant Id.at 16. DOHMH also responded that it"mwgnizcd deficiencies in its mrrenr system of OCR scanning oxhnology that depends on handwriting recognition rechnology to If."Id. aay. /d-113. On,. Jeremy Smerd, Tick, Tick, Tick for ffiaonne's Lq g, Croons N.Y. Bus., Jan. 1s, nos , hop://w w.crunsncwyork-coMarticl4mm�ou5/polidcs/3ou59963. 6sz FUDGING THE NUDGE grade, required to be posted in a location visible to passersby.�3' In principle, the conversion was simple: fewer than 14 points resulted in an'A; 14-27 points resulted in a'B';and 28 or more points resulted in a'C.1232 In practice, the implementation was more complicated. First, DOHMH introduced a system of "reinspection" for grading purposes. If an Initial inspection resulted in fewer than 14 points, a restaurant received an 'A.' If, however, the restaurant scored above 14 points on the initial inspection, DOHMH scheduled a reinspection to occur roughly a month (and"no sooner than 7 days"'33) after the initial inspection." Such reinspection were, in theory, plenary inspections conducted by a new inspector, the score of which would determine the restaurants grade.233 For example,a restaurant scoring zo points in the initial inspection could be issued a final grade of'&' 'B,' or 'C,' depending entirely on the score upon reinspection. As before, restaurants scoring 28 or more points on any inspection were, in principle, subject to compliance inspections'3' Second, the xoro revision changed the timing of the inspection cycle. For any restaurant receiving 28 or more points on either the initial inspection or reinspection, "[a]n initial inspection commencing a new cycle shall be conducted go to 35o days after the" last full inspection.�3' For any restaurant receiving 14 to 27 points on either the initial inspection or reinspection, "[a]n initial inspection commencing a new cycle shall be conducted iso to 210 days after the last full inspection.i}8 All remaining restaurants received initial inspections roughly once a year.i3" 231. See 137 CITY REG. 16o8 Qnne is,xoro); 137 CUT Rae. 698 (Mar.23,nom);Malso N.Y., N.Y.HEUTH CODE S 81.51(c)(zou). aha. 137 Cm Mr.1607(June q,2m0). Id. 234, Sea Reer¢urNu Lew Grading: The First Year, N.Y.C. DEPT OF HF,u.TH & Mstrrn3. HYGIENE 3 (zou), hap://www.nyc.gov/h,.Vdoh/d.—A-d,/pdf/rii/rrsmr=-grading-i _yeti-nponpdf(noting that reinspection oMaM 'about a Mundt la ,C). in the interim period,resraurmts were required to con inne to post a prior grade,if any.See 137 CITY REa 16o8(June IS,xoro). ars. In Ins Mgclu, such reinspection ale liniad in that do y only examine violas ince died during dhe initial inspection. This difference was widely contested by it. New York Aesdaurent Assoeiarion.See,e.g.,Conurning Leiter Grades Aoposal,Hmririg BggbO rhe N.Y.C. Dept iF Hed1h &MOW Hygim2 (Feb. S. 2010) (stamntenr ,of Ruben Bue lmun, N.Y.C. Counsel,N.Y.State R I.Ass'o). n36k SO 137 CITY Fee-1607(June 1S.2010). 237. Id- .,B. d23a. Id 239. Id. 6x3 THE TALE LAW JOURNAL 122:574 2011 Third, restaurants that did not receive an 'A' upon reinspection were provided the option of posting either the final grade (of'B'or'C') or a"grade pending" card until a hearing in front of an administrative tribunal."" (In practice, most restaurants choose the latter.) Taken together, these three complications were quite favorable to restaurants. Restaurants had a second chance to achieve a higher grade, did not in fact have to disclose a poor grade until an administrative resolution, faced shorter actual disclosure periods with poor grades became of the shortened inspection cycle, and gained greater certainty as to the timing of initial inspections and reinspection. The aom changes also included some minor changes to the scoring system."" (Appendix Ca shows that the types of violations documented are comparable pre- and pose-grading.) Yet DOHMH itself stated, "[T]he HCALh Department has not changed the way it conducts inspections [with Jan.Id az,608.The Rules arc actually hmd."r abont the impact of.dutin ot.dire hearings for a notice of violation issued in rhe initial inspection.Notice of violation can be issued in any scored impavon,see id at h6o7,but a grade is required to be posted upon resolution of an administradw hearing for a wirupection, id. at i6o8. It is unclear, for example, what ramifications an administrative hearing that reduces below 14 pmras an initial inspection score—which would retrospectively obviate the reinspection—has on die grade, wlridi is supposed to be based on a reinspection when conducted. Compare ut az t607 ("The Deparrment shall issue a letter grade of'A'w any establishment that receives fewer dun 14 points on either the initial iiupection or reinspection in a cycle.')(emphasis added),with id. a Lune ("[I]f the establishment docs not appear at rhe Adminisnative Tribunal . . . the establishment shall,on die date of the hearing,post the letter grade card provided by die Depamnent or the neirupeomm.")(emphasis added).In June amt,die Mayor co afened die responsibility for holding administrative hearings from rhe DOHMH Adminismtive Tribunal to the New York City Health Tribunal in the Office of Adminismtive Trials and Hearings(OATH).Compare id (requiring adjudication by OATH,of which the New York City Health Tribunal is a mmponum), rvith Exec. Order No. MS, Transfer of Curtain Tribunals and Adjudirarmy Functions Consistent with Mayor's Committee Report(]new 8, iou)(requiring adjudication by the New York City Health Tribunal). air. For example,a aB violation(the second entry in Table;)was amended m allow conditions in change when the same food was out of die holding temperam a in different areas. In too,a s condition It violation was, found for"[r]wo hot food items out of«mpenmrc. Example: 8 clhidien wings and cooked rice." Imprxrion Scoring System for Food Service Esmbhshmems,copra note ahq,at u.In 2010,a a condition II violation was found for"[t]wo hot food items out of rempetature or die same type of food out of temperature in two different areas.Example:one my of chicken wings and a pot of the held at n5-P;or one tray of chicken wings on die swam table and one tray of chicken wings in the food preparation area held at J,-F." IVhrt To Expea When You're lopertd:A Guide for Food Se.vba Opemturs,supra now azo,at 8. In addition,some"Other Critical[]"violations were deleted, as was a violation for "food intended for consumption in contact with toxic manual"dart duplicated other renomination violations.N.Y.C.Dept of Health&Mental Hygiene,Notice of Adoption of a Role Repealing aM Reoodifying Chapor 23 of Title 4 of the Rtes ofrhe City of New York,H10TV Rae.16.6-u(Jane is,amu). 6M FUDGING THE NUDGE grading]. "' The timing of the intervention was sharp. In the beginning of July sato, restaurant inspections slowed to a halt as inspectors underwent training for the new regime. Every restaurant inspected after July 26, 2oto'o would receive a full initial inspection for grading purposes,although it would take over a year to complete the first grading inspection for all New York restaurants.'"" 4.Internal Assessment After the first year of grading, DOHMH released a triumphant assessment."n First,based on a survey of 502 respondents, it reported that 70% of New Yorkers have noticed grades in restaurant windows and that 88% of those who noticed them consider the grades when deciding where to eat (a stretch in interpretation).'""° Belatedly, Zagat reported that only t% of respondents (albeit in a convenience sample) would eat at a 'C-graded restaurant.'"' Second, the report concluded that grading "Has Contributed to al=. Leser G�gfor Sunnily Impatiom: Wha[h Mes"J"you-nousaitd Conswnas,N.Y.G. DEPT OF HEALTH & MENrA HYGIENE 2 (Jmu' 1010), 11".//W .Llyc.gov/hmWmtll /dowsdoads/pdf/resuurmc-gradulg-faq.pdf lhemnufer Grading:What It Mans. an. Tedmically,die sides were published on June IS.20 to,and became effective July S.aom. So 137 Cm'REe. 1,606 (June i5. aom); Comm r of Health Se Mend Hygiene,N-do'of Adoption ofAmendmmrs 0 Chapter 23 Of Tide a4 f the Auks of the City ofNew York,DEFT of HEN.TH & MENTAL HYGIENE 1 (2010), hap://wwsv.nyc.gov/h=Vdoh/downloads/pdf /nodW2oln/nodm-chap-23-fide-M-die-mlu- E-nyc-conxcfion.pdf (noting effective dace Offaly S.Imo).Grading's implementation,however,did not begin until July 2y,1oa0.See How We Smrc and Grade, N.Y.C. DEPT OF HEN.TH & MENTAL HYGIENE a (aal2), hap.//wwsv.nyc.gov/ht.Vdoh/dowliloads/pdQriyhow-we--ro -grade.pdf (noting that "lilmPsm io 1,before ply 1I,aom"are ungraded). aµ. See Grading. What It Mons,supra nore zt2,at 2. sag. See Restaurant Lemer Grading:The First Year,Myra note airy. all.See id.at I.The report,ptoblerma ally,does not disclose the undedying survey instalment. Upon obtaining the survey instrument, die interpretation is arguably imprecise.The 88% figure is the sum of36%who"always,"2t,%who anameofdu time;and23%who'W=of die lime""consider fire letter grade, m "esutng die respondent has seen the grades(me of 70% of respondents). See Banmh Coll., Sch. of Pub. Affairs, NYC DOHMH Restaurant Program Evadualion Poll (July IOU) (unpublished survey) (on 41e with author).A more aautair statement would be that six of ten New Yorkers have seen and mtasider rhe letter grades at Ieansome ofthe rime.See I& aur. Christie Rotondo&Rich Schapiro,Ni,Yorkers Won't ILU of a Ramuran[with a'C'Grade from Heuhh Depawaau, Zagar Survey Says, N.Y. DNLY NEWS. Oa. 5, 2011. hap://ardcle.nyda0ynews.com/2o U-I D-o5/enmrtammerat/3oa616,_1_2agat-le-liermrdin -c-rated. 625 THE YALE LAW JOURNAL 122:574 2012 Significant Improvements in Restaurants' Food Safety Practices.""a For example, DOHMH touted that 69%of restaurants have achieved'A's,rs'a fact that by itself says little about the impact of grades. Third, DOHMH reported that restaurants "improve between their initial and re-inspections,' noting,for example, that 38%of restaurants souring between 14 and 27 points in the initial inspection end up receiving an'A! in the reinspection.''°What this glosses over is that 39%of restaurants receive an'A' in the initial inspection. o The process could be entirely random, with a two-fifths chance of an'A'in any inspection. Fourth, DOHMH reported that restaurant training has increased and that inspectors have conducted unannounced inspections for grade card posting compliance, with over one thousand violations cited.'''The existence of these violations, however, also suggests that restaurants are flouting the posting requirement. None of this DOHMH evidence amotmts to a plausible assessment of the grading system."J B.Inspections Data To study the grading system more rigorously, we use a publicly available dataset from December 2011, which contains inspection results for all existing restaurants in New York. (Our data analysis actually began with a dataset from July 2orl, but for simplicity of exposition, we focus on this most recent version.) The primitive units are 495,568 violations (or nonviolations where an inspection resulted in no citations). Each unit contains information about the restaurant, permit number, inspection date and time, violation codes, action taken by DOHMH, numerical score, and grade assigned. Using the date and permit number, we restructure this data to the inspection level,"* creating a dataset with 126,938 scored inspections for 23,153 restaurants. 246.Sea Resmurara Lever Grading:The Four You,supra note zm.at 2. aces.Id at a. rso. Id.at s. rn. Id. rsr. Id.at 5. as, DOHMH's is month report, Resmuraut Grading in Nna York City ar us Mondrs,N.Y.C. DEPT of HEALTH &!Vew L HYGIENE (2012), htrp.//wWW.nyc.gov/htmVdob/dowNOa& /pdf/riVmsraurant-grading-rs-monde-repoa.pdf, similarly reports otdy superficial claims that are difficult to understand or replicate. zsa Most compliance inspections are recorded as mos coed in the dataset. For the analysis of compliance urpections, we of course include these in our analysis. For seventy-fora reetaamm-inspection dares,action codes and scores are not homogeneous.We omit these reseanrams from our analysis. 626 FUDGING THE NUDGE DOHMH deserves much credit for making this data available. Most jurisdictions have websites that allow users to look up individual restaurants (as we did to download the San Diego data), but the full microdata are rarely available in direct, machine-readable format(see Table Q. That said, the DOHMH data have distinct limitations. First, DOHMH omits information for restaurants that are no longer in existence. Given that the restaurant market is quite dynamic, with the best studies suggesting that approximately one-third of restaurants fail within one year and two-thirds within three years,"'this means that we have much less information from the early years of the observation period. Second, although our microdam in principle cover the same inspections as the data DOHMH makes available online, they do not include the type of inspection conducted (e.g., initial inspection, reinspection,compliance inspection).Third,the online data appear to be locked from access outside of the Greater New York area. California residents (or researchers), for example, cannot access the inspection-speck data online."`This makes it difficult to augment our existing mictodata with the type of inspection by automatically querying the online site.We solve this problem by developing a finely tuned classification algorithm that leverages information in the New York City Rules and DOHMH policy to classify inspection types (spelled out in detail in Appendix 13). For a random (cross- validationan) sample of five hundred inspections coded manually from the online data,our algorithm classifies 97%of inspections correctly. Fourth, the reason it does not appear possible To classify inspections perfectly is that there are a number of internal inconsistencies within the DOHMH data In some instances, for example, the letter grade is inconsistent with the numerical score. Appendix A more comprehensively documents data integrity issues we have uncovered, which are severe but limited enough in scope so as not to affect our findings. Last, none of the DOHMH data dearly represent whether an administrative hearing occurred and, if so, whether the score was reduced at the hearing. As best as we are able to determine, DOHMH simply overwrites the score, grade, or both. (Changes in how DOHMH accounted for hearings may explain some of the internal inconsistencies of Appendix A.) We examine the effect of administrative ass Su H.G.Pusa a al., Why IZStn mm,FWI,46 Cow+rlL HM,,&RFSrAUnNir ADMN.Q. 304,309&ex.1(a0a5). ass. We auempted t0 occas dke inspection-specific data from multiple Cafitomia-bued maduna on multiple operating sysrems on multiple days and with multiple browses.Tlx ChiefTeehuology Officer ac Stmf rd UW School was nor able to solve the problem. ase. By cross-validation,We mean that dus sample of five humdred impermuo war not used to develop he dasslfie.mm algodd:m. Gal THE YALE LAW JOURNAL 122:5 74 aria hearings in Appendix C.3, by using multiple versions of the dataset we obtained privately from January 2010 to January 2012. These multiple versions enable us to track how an inspection is recorded before and after an administrative hearing, as differences in how the same inspection is represented across versions are most plausibly attributable to hearings. Figure 6. INSPECTION TYPES IN NEW YORK Ralosing Compliance Other Pre-pcmnic Reopening Rcinsponion Initial 0 10,000 ao,000 30,000 40,000 Nunnber The vast majority ofscored inspections are initial inspections and reinspection. Figure 6 plots the distribution of types of scored inspections from our Classification algorithm. For the moment, these data exclude unscored compliance inspections (the bulk of compliance inspections),which we return to in Part Vl. The vast majority of scored inspections are either initial inspections or reinspection."s ala. For completeness,the other kinds of inspections are As follows.After a restaurant is shut down,it must undergo an impection m reopen.If such an inspection is successful,it mums as a"reopening inspection,"triggering a subsequent initial inspection;if wt.it counts as a 4ed0sing impe uim," keeping rhe mseauranr closed. "Pre-permit hsspe<tion" arc conducted for start-up restaurants prior in operation, and arc followed by an initial inspection.To be precise,we classify as an initial inspection an initial operational pre-permit inspecriw,because the subsequent inspection typically is a reinspection(e.g.,Aunt Rosie's Coffee Shop And Diner). We classify as a pre-permit inspection an initial nonoperational pre-permit inspection, because the snbsegnea inspection is typically m initial insspecdon (e.g.,Crown Fried Chicken,3q 86 Broadway). 6x8 FUDGING THE NUDGE C.Empirical Result t. Grade Variation Unlike San Diego, New York exhibits considerable variation in grades. Roughly 61%of restaurants inspected each month receive'A's.The mean score across all inspections is 19 points (standard deviation = 15). Figure 7 plots the proportion of restaurants with specific grades assigned in each month's inspections in the top panel (i.c., the "flow" of grades), the overall proportion of restaurants with each grade (based on the most recent grade) in the middle panel (i.e., the "stock" of grades), and the raw score in the bottom panel. The x-axis represents the month of the inspection and the y-axis represents the score or proportion of restaurants with a specific grade.The proportion of'A's assigned in any given month (the flow) remains stable across time. The proportion of restaurants with grades pending spikes upward in the last months of the observation period, due to pending administrative hearings. Mutually no restaurants that were inspected in December 2011 actually posted grade signs of'B'or'C,' as indicated by the dip in the dashed black and solid gray lines. This simple time trend illustrates how the "grade pending" option makes the system more palatable to restaurateurs.Most consumers arguably do not know how to interpret a"grade pending"sign.The delay for a hearing and faster pace of inspections for low-graded restaurants mean that grades of B'or 'Care pasted for only short durations. Despite the fact that the proportion of restaurants receiving 'A's in any given month (the flow) is roughly constant, the stock of Ws (from the most recent grade) increases over time, as depicted in the middle panel. This stock- flow difference may be mechanistically driven—even with no general improvement in sanitation practices and random scoring. Because the inspection cycle is shortened for lower-scoring restaurants and more protracted for higher-scoring restaurants, the stock of'A's will increase over time, even under random scoring. bzq THE YALE LAW JOURNAL txx:574 aora Figure 7. RESTAURANT GRADES OVER TIME IN NEW YORK m o GRADES OVEI TIME C G ter. O NnING O o MOST'1sECEN"I GRADE c 0 0 0 0 u RAW SCORE v y 12010 12011 Time Pending grades are issued when(r) neider the initial inspection nor die reinspection resulted in an'A,'and(2)an administrative appeal of die inspection is pending.T7nese pending grad.are conversed res grades once die appeal is resolved. Moreover, while DOHMH proudly cites over one thousand violations for failure to comply with grade posting,�i9 such violations also suggest that disclosure can be evaded. Figure 8 shows one example of formal compliance with posting requirements that likely avoids informing consumers of the health inspection result.Writes one commenter on the New York Times website: Helpful tip to restaurant owners who happen to get C grade [sic] whether they deserve them or not: the bright orange C letter grade signs fade rapidly in direct sunlight so while you have your grade as° See Aesmur¢ns Lnrer Crsdirg:The Firse Year,suprn nose xg4,at 5. 630 FUDGING THE NUDGE pending signs posted—take your C letter grade home and hang it in direct sunlight,in about 7 days it will be very faded.i ' The New York Daily News reported chat several pizzerias,a bagel score,and a Dunkin' Donuts franchise registered as supermarkets or wholesalers, which fall under the jurisdiction of the state Department of Agriculture, to evade grading requirements.` Notwithstanding these forms of selective disclosure, New York restaurants exhibit genuine grade disparities. On that measure, New York appears to provide more meaningful information to consumers than San Diego. Figure 8. DISCRETIONARY GRADE DISCLOSURE'°' b Tlus ..,.t was assigned a 'B' grade and pvned the grade card at die door as -specified,at least formally,by the Nev,York Guy Rates.Phoro by Zach Seward for the Walt Srreer Journal,repruited with photagrapher's pumisssoa aso.danid, Comment . McCabe, stip. Hare 70 (Jan. a,, 2011, 10:49 AM), http://fiveddrryeighlblogs.iiydmcs.conVaolgoi/i,/grading-new-york-restaurants-whaw -in-an-f?commentsapermid=6. a6,. See Iteuwn Blau&Simone Weicliadbaum, How Eotena Avoid Health Dept.��o Grades, N.Y.DAILY NEWS,)an.9,aor;hap: ews //arucia.nydailyn .com/Iola oo-o 60 668 i letmr-grades-andrew-dgic-restaurants. aha. Aaron RtakoR,Barnum.Make Bar Our of'B'Grade,WAu.ST.J.:METROPOLIS(Sept.17, 2010. 4:04 pM), http://blogs,u j c nVmetropolis/20so/09/9/rumur t-makes-du-best -0171>grade. 631 THE YALE LAW JOURNAL 122:S74 1012 2. Grading Changes Scoring How have inspections changed with the onset of grading? Figure 9 plots the distribution of all inspection scores pre-grading on the left panel and post- grading on the right panel. The dashed line on the left panel indicates the threshold for failing a health inspection, while the dashed lines on the right panel represent the grading thresholds. Most strikingly, sharp discontinuities exist a each of the thresholds. Prior to grading, 1,424 inspections resulted in a score of 13, and 1,784 inspections resulted in a score of 14.After grading, 3,923 inspections resulted in a score Of 13,and 1,416 inspections resulted in a score of 14(p-value for the difference in proportions < 0.0001).i13 Figure 9. INSPECTION SCORES IN NEW YORK vu-cxnnns eosT-caanns o` �Poilal A 8 C 6 r c � Y A I 0 1. ao 30 40 so 60 o In ao 1 4- yo 60 Scare Some These histograms depict inspection scores for all scored inspectors;before grading in the len panel and after grading in the right panel.The threshold of 2R points in the pre- grading period resulted in compliance inspections dun could lead ro a shutdown.The threshold of y points in die post-grading period dcrermincd the difference between an 'A' and W grade. The threshold of 28 points could continue to result in compliance inspections, but also revdred in a 'C' grade. The tnesholds exhibit sharp discaminoides. Given the slight scoring changes and reinspection system first introduced in tom, we now examine scoring and violations over more fine-grained time periods,using our classification algorithm to classify types of inspections in the pose-grading period. Figure to plots sequences of histograms as time proceeds a63. The p-value is calculated from a Fisher's exact test applied to a two-by-two contingohcy table with rows representing stores of 13 and 14 and columns representing the pre- and post-grading periods. 63a FUDGING THE NUDGE across rows: the pre-grading period is split between the period before and after July 3009 (when DOHMH dropped documentation and administrative violations), and the post-grading period is divided into quarters containing roughly an equal number of inspections. The columns represent initial inspections, reinspection in the post-grading period, inspections resulting in closures, and counts of violations. The gray vertical lines indicate thresholds (for failure pre-July aoro and grades post-July aoio)and the short black dashes represent averages. Several trends emerge. First, scoring during the pre-grading period and first post-grading period appears comparable, which suggests that changes in the scoring system had little immediate impact on the conduct of inspections. Second, as time progresses over the quarters,we observe considerable shifts in the distribution of scores in both the initial inspections and reinspection.The discontinuity at 14 points emerges for both, but much more sharply for reinspections. Reinspection, which typically happen within a month, are disproportionately responsible for the shift toward borderline 'A's and 'B's. Third,the number of violations cited in initial inspections(the gray histograms in the right column) also appears comparable over time, but reinspection generally result in fewer violations..w Last, the scores of inspection resulting in closure of the restaurant cluster sharply to the right, as we might expect. (A considerable number of inspections result in a score of o, which may result from inspectors shutting down the restaurant without formally tallying the score.) aµ.The diKerencc of mi%Wy o.6 violations is statistically significant (p-value from o-tesr <ovoor). 633 THE YALE LAW JOURNAL 122:974 aura Figure to. EVOLUTION OF THE GRADING SYSTEM IN NEW YORK PRE- INITIAL CLOSINGS VIOLATIONS GRADING (-'49,-S) (.`998) Pre- Jdy2009 I post- Julyzo09 POST- GRADING (n=4xsx6) (n-z9.zaa) (nm,9P) Rdrspecaion First quarter tuitul Second '. quaaer I i Third quarter Fourth quarter o 02 w 2 4o 6o o 4o 6o 0 4O 6o O 4 8 u Score S. Score Number The Lap two Laws depict inspection scares prior to the grading system The bottom rout sows depict inspection scares as time progresses from the first to dee fourth quarter of the post-grading period The first three columns correspond to different types of inspections: initial inspections that sun a scoring cyde in the first column, inspections for grading purposes in the past-grading period in the second column, and inspections ranking in DOHMH closing the restaurant in the ddrd column. DOHMH an shut down a MOILI am for "Actions and persistent violations or uncorrected public health harandan265 evco when its score does nor exceed 28.The last 265. 137 CITY Ree 1607(June ray,IGIO). 634 FUDGING THE NUDGE .Into,, presents he ontober of violations for initial inspecdons and reirepectimu. Vertical gray lines indicate applicable ho.h.lds: (t) a8 points for a F tled iisspeaion pre-grading or a 'C-grade post-grading and (2) t4 porn,, for an 'A'-grade polo- grading. Short black dashes indicace the average score for the period and type of inspoenon. As with San Diego, from this data alone, it is not possible to infer whether the system is working effectively. The discontinuity observed a the threshold between 'A' and 'B' grades, for example, may result from restaurants targeting the threshold by cleaning up just enough to achieve a better grade. If anything, however,average scores are incasing over time(see the black dashes in Figure 1o),suggesting no general improvement. 3. Scoring Inconsistency If New York is unlike San Diego in that it has actual grade variation, how does it perform in the consistency of scores? One requirement of targeted transparency is that the underlying information content must be meaningful. To examine this,we compare initial inspections across inspection cycles in the post-grading period. Initial inspections are the closest to random inspections because their timing is the least predictable and are therefore most likely to provide an unbiased measurement of restaurant sanitation.i ' As DOHMH states, inspections across cycles "are an indicator of restaurants' typical food safety practiccaiieo (For robustness, Appendix C.z shows that the lack of consistency persists regardless of the type of inspection examined.) Figure it plots the score of the first post-grading initial inspection on the x- axis against the subsequent initial inspection on the y-axis. Because the post- grading period is only 1.5 years long, we observe 14,552 restaurants (roughly 63% of the sample) undergo multiple initial inspections. Each dot (randomly littered for visibility) represents one restaurant. Unlike in Figure 4 for San Diego,the mass of data looks essentially random. Roughly z5%of the variation in San Diego inspection scores can be explained by the previous cycle's scores, but prior scores in New York explain less than 2%of score variation.Of course, restaurants scoring poorly have an incentive to improve for subsequent inspections. To account for this, the bottom panel focuses on the subset of restaurants that received an 'A' in both initial inspections. Again, there is no substantively meaningful correlation across the cycles. acs.See Hatfield&Seiver,seyva note Ira,at 23("(Tllu,eeiraspeetioa grade is less likely to be an unbiased indication ofongoing operations."). nbt. Restaurant Lerner Grading The Fine Year,supra nice 234,at 4. 635 THE YALE LAW JOURNAL 122:574 2012 Figure it. LACK OF CONSISTENCY IN NEW YORK FULL RANGE > f 6o 50 40 3° ao to 0 C•C)e I 'A ORANGE Y p g 6 4 a 0 cye[e 1 Each dot rapta cess, the score a relons.,received in du first initial inspection after grading was institneed on the x-axis and the snbsegnrnt initial inspecdon on the yaws. For visibility, observations arc randomly littered.The top panel depicts the range of observed scores(..red at 60)and the bottom panel dep,.neswnrants rerxiving'A' 636 FUDGING THE NUDGE grades in body cycles.Less dist z%of die venation io inspecrion sores is predicied by prior inspection sores(see R's in die lower right hand comers).Unlike in San Diego, inspections do not appno to measure mesiurgfid Systematic ssutntion differences amongst restaurants. It doesn't take long to discover the lack of consistency upon sampling individual restaurants on the DOHMH site. Per Se,a three-star Michelin-rated restaurant, received 23 points on its first post-grading initial inspection, 8 points (and an 'A') on the reinspection, and 41 points on the next initial inspection.i ' Ma anon's Falafel, a popular, hole-in-the-wall falafel shop in Greenwich Village, received 59 points on its first post-grading initial inspection,5 points(and an'A')on its reinspection,16 points on its subsequent initial inspection, q points (and an 'A') on the reinspection, 27 points on its third initial inspection,and 26 points (grade pending)on the reinspection. The considerable noise in New York inspection scores—particularly when compared to their relative consistency in San Diego—means that grades are not particularly good predictors of future inspection scores. A 10-point increase in one initial cycle is associated with a statistically significant 1.4-to-1.7-point increase(at )5%confidence)in the subsequent initial cycle.Figure 12 illustrates what this substantively means, plotting the distribution of scores in the subsequent initial cycle given an initial inspection in the'&' 'B,'or'C'range. If a restaurant receives a score in the'A' range, it has a 37%chance of getting an 'A' the next time around; a W-range restaurant has a z7°% chance, and a 'C'- range restaurant has a ao% chance. To be sure, repeat initial inspections do exhibit some degree of correlation, but grades as disclosed provide customers a false sense of certainty about the restaurants current sanitation pmctice.i 9 Unlike in San Diego, the distributions (particularly in the 'A' and 'B' panels) exhibit strong similarities. :68. After die first draft of dus Article, rhe Nnn York Pan reported that the mstager of Per Se, whose owner liar ties to Mayor Bloomberg,called DOHMH to improve the saturation sore of an inspection outside our observation period.See David Seifinan,A-Ro ed'Anvtfor'Connctrd' Eatery EaUaun.gTr lafimn:Se Whoa?'Comueted'Eatayi Gmd Bmnp,N.Y.POsr,Mar.3,aoiz, http://wwsv.nypost.mnVp/news/loot/marJutta,Vmt c_=ist_T4Q ciSEBgbgHozf[Jal. sy. For some discussion ofd c limits of W n a measure ofsubsrandve wa sistency,we Dsud E. Ho,RaostaEng Puniriw Danwges Evidence, 166).INSTITUTIONAL&TNnoaanrat ECON.z7 (aoio). 637 THE YALE LAW JOURNAL raa:574 aole Figure Ia. LACK OF PREDICTIVE POWER IN NEW YORK cVRRLNT'A RANCE lossun T'a"MNLC CV n0.CNT'C MNGE q O c nTTt—t—'t O r---STC-6 O A 20 40 6 40 6. 4A A 4n 6A o Score in next cycle A some in nsr cycle Sm.in next eysle Each panel depicts cycle a seems given the restaurant grade in cycle J.For example.die left panel depicts die smm distribution in cycle a,given drat a mucaumt received an'A' in cycle 1.Although them are shifts, rhe distributions are not sharply distinguishable from a sub nermic perspective.For visibility,seers are reasoned at 60. V. EXPLAINING THE SCORING DIVERGENCE What explains the sharp differences in inspector scoring practices between San Diego and New York? After all, the inspections systems share some considerable similarities: both (1) follow the FDA model food code (as do many jurisdictions); (a) employ a point scoring system for substantively comparable violations with reinspections for grading purposes; (3) engage in similar hiring practices for health inspectors; (4) operate in relatively diverse restaurant markets; and (5) visit establishments a comparable frequencies.ijo There are of course other factors that may explain the difference, such as greater dynamism in New York's restaurant market, the shorter amount of time its system has been in place, and the penalties the two jurisdictions impose for poor performance(which we address in Appendix D). We explore one alternative explanation. Perhaps targeted transparency has emphasized the demand side of information but ignored the supply side. Put differently, targeted transparency prescribes simplifying information to enable no. The average number of days between scored impectimhs for an establishment is ger (standard deviation= 139) in New York and 184 (standard deviation= u6)in San Diego. Although Table i indicates that die minimum number of amoral impecnmu is ane in New York Led ram in Sahr Diego,those minima refer re the number of inspections for the lowest- risk types ofesabhshernam. 638 FUDGING THE NUDGE comumem to act in a sophisticated way, but perhaps New York fails in simplifying the inspection process sufficiently to enable inspectors to score in a consistent way. Differences in the way inspectors conduct on-site visits might explain the noisiness in the New York data To investigate this possibility,we studied in detail the inspection processes, documentation,and scoring worksheets used in San Diego and New York.Our goal was to formalize how each jurisdiction might Great the same underlying behavior or condition. We used our materials to map u5 New York violations"' to fifty-two San Diego violations, allowing for any kind of mapping (e.g., one New York violation might match several San Diego violations). In some instances, the marches were straightforward.For example, San Diego's violation of "returned and reservice of food—r' matched New York's violation of"unprotected food re-served—n Other instances, however, required more detailed parsing of the materials. "Food handler training"in San Diego,"`for example, requires that any employee in contact with food have a valid food handler's card New York's requirement of a food protection certificate, however, applies only to supervisors,"s and therefore is not a substantive match. Figure 13 presents the results from this comparison, focusing on scored violations that are cited at least once.The rows represent scored violations and are sorted by the frequency of citation in New York(plotted on the left panel). Each square represents point values that can be assigned to that violation.Gray squares are general (or minor) violations, and black squares denote critical (or major) violations. For example, the top row in the New York scoring panel represents a violation for improper maintenance of a non-food-contact surface. In New York, this ioF violation is a general violation, denoted by the gray boxes, and may be assigned z, 3, 4, or y points. The lines connect New York's violations to a comparable San Diego violation. For example, the horizontal line in the top row matches New York's ioF violation to San Diego's #33 violation for an unclean non-food-contact surface, which is a minor violation scored at one point. .p. One hundred fifteen is more than dw number of violation nomd in the caption of Table 3 primarily because ofmoomed,administrative violations added since ants. nr. Retail Food Funky Openus,s Guide,supra none n3,at tz. tri. What To Expeo Whoa You're Inyozed:A Guulefor Fwd Soviet Operamrs,supra note zzo,az 13. era. Rearil Food Facility Opentror'e Guide,supm note 213,at 8. »s. What To Evpeor When You're lmpoud:A Gmilefor frond Service Open,supra ROM>zo,at u. 639 THE YALE LAW JOURNAL 112:574 2012 Figure 13. SCORING IN NEW YORK AN D SAN DIEGO is I I 3i�°iF1 so �11,19 ' Y €3 ' III 110 .� I 8 I I III I 1 III '. L� Y 8 G [ II ' I I ' I�II�If�71I�j;aiiP�'v'yv .'s aey— RM, M i g a[e: IIjI I' gppyp '?4ttt qa = a 3 egijpgaj3� 9 1?°e i$ i@g3 � IIp:'gt'i g B3itv Eee '°. gii i�Ei oc ..it9i3 Si 3�3�g .t E �� � i � i3 •• 4 • J S 8 ei s 640 FUDGING THE NUDGE The left panel depicts the&cquency ofviolatiorss,hued on New York data in de post- grading period. For example, improper construction of a non-food-contact surface is die most frequently cited violation.The"Nev,York Sorting"mid•San Diego Scoring" panels depict the points that an inspector on assign TO such violations in Nev,York and San Diego, respectively. For trample, a New York inspector can assign z, 3,4, or 5 points for improper maintenance of a non-food-contact surface violation,while a San Diego inspector can assign t point.Black indicates a"aicicX or"major violation and gray indicates a "general" or "minor- violation. The vertical lines represent the thresholds for an W grade in the two jurisdictions. The light gray lines connecting these panels march up substantive violatimss. New York disaggregates classes of violations more finely than San Diego and uses a much wider point range(relative w the grade threshold)for most violations.Both factors arguably increase lmpecror discretion. Two findings emerge from Figure 13. First, New York inspectors have a larger set of violations to score.While New York inspectors can cite sixty-eight possible scored violations, San Diego inspectors can only cite forty-eight.This does not mean,however, that San Diego inspectors ignore underlying behavior that is cited in New York Most New York violations ran in fact be mapped to a San Diego violation, as indicated by the connecting lines between the panels. The difference, instead, is that New York disaggregates classes of violations more finely, as can be seen by the fact that a single violation in San Diego is often mapped to multiple New York violations. For example, a violation of "[n]o rodents, insects, birds or animals" receives either a or 4 points in San Diego."New York,however,records separate violations for(t)"[e]vidence,of rats or live tars," (z) "[e]vidence of mice or live mice," (3) "[I]ive roaches,"and (4) "filth flies,"each of which can be scored 5, 6, 7, 8, or 28 points, depending on the amount of evidence.''Thirty"fresh mice droppings in one area" result in 6 points, but thirty-one mice droppings result in 7 points."' Other "[I]ive animal"violations are assigned 5,6,7,or 8,but never 28,points."" Second,New York inspectors retain much more discretion in the potential range of point scores for the same underlying violation. For instance, general violations for plumbing not being "properly installed or maintained" (1oB), pesticide use "not in accordance with label or applicable laws" (8C), or simply "other" issues (qqB) can carry anywhere from z to 28 points."o In principle, n6. keuril Food FaeiGry Opemturs Guide,supra note 113,at 6. an. LVhu To Expre Whin Yoti relmprmd:AGsudefar Food Savin Opemrors,mpranoezxo,atnq- n5:rmlrupation Swung System farFrood Service Fsud!vhmw,suyra note ss4,at;. nx. What To Expex When You'relWu d:AGadefor Fwd Sow Opnwon.supmiwwe o,atrq. e,g Id.at q. .so. Id.ar an,ao,z;. 641 THE VALE LAW JOURNAL rai.S74 2012 condition levels are meant m capture the severity of a violation;"' in practice, they afford more discretion to inspectors, thereby potentially undermining the goal of numerical scoring to make the inspection process more objective.Relative to respective',PC-grade thresholds (the vertical lines), New York inspectors have much more discretion than San Diego's to amazs violation poi 's' points Could such differences in the inspection score sheet matter? There are several reasons to think so. First, New York inspectors do not appear to be specializing exclusively in restaurant health inspections. In the aooq audit, the Comptroller found that only sixty-seven inspectors (out of some 16o at the time) actually conducted more than one hundred restaurant inspections per year.�S3 Second, given the salary levels, it may not be easy to retain talented inspectors, leading to tumover in staffing. Third, the inspection score sheet may uniquely matter for the twenty new inspectors hired by New York after the grading system.Each of these new hires would have little prior experience scoring inspections, thereby exacerbating variability across inspectors. Fourth, supervision, az the Comptroller's audit showed, is lackadaisical."O Given the sheer number of inspections,New York's scoring system may be too ambitious m induce any consistency across inspectors. Fifth, because the scoring system was only introduced in zoo;, inspectors have at most eight years of experience with the system. Last, the design of inspection worksheets appears to matter elsewhere. In Santa Clara County,for example, one inspector inflated swaths of Palo Alto restaurant stores by accidentally checking off major and minor violation boxes for the same underlying violation.That inspector in Santa Clara County reported 442 major infractions;another inspector reported none.'ss It is of course possible that other differences in institutions(e.g.,the degree to which inspectors specialize in food safety)—not the design of the inspection scoring process—explain the divergence between San Diego and New York. Los Angeles, however, uses scoring comparable to San Diego and exhibits similar consistency across inspections, but its inspectors specialize to an even lesser degree than those of San Diego (zlo individuals functioning as the aa.. See MCSwane tt al.,Supra Sore 127 at ;44 (discussing proposals to lunge the mo-point score worksheet of the 1976 model food code to allow for weighting by severity of violations). ,a2. This remains the case even if we ignore point values of aa,which,one might argue,simply represent the ability that most jurisdictions have to shut down aa esrablishment for a serious public health harard. :93. See 2009 Awn,.rupm nom 204.at 13-14. 294. Id- 29S. Sec Steve Johnson, Errors in Food Wry Checkups Inspeaw,u; Va71010r0 i^ R Aimrmr A64aI iiisoriom Due es LAok of Uniform Statdards,SAN JOsa MEaeuav NEWS,Aug.eco,2000,at LA- 64a FUDGING THE NUDGE equivalent of 106 full-time employees). Until designs are tested, we will not know for sure.Nonetheless, the substantive comparison of the scoring process (and evidence across these jurisdictions) suggests that there may be such a thing as too much information,not only on the disclosure side, but also on the production side. VI. INTENDED AND UNINTENDED EFFECTS We return now to New York. Its implementation of grades during our observation period allows Us to potentially assess some effects of grading (subject to the caveats in Section ILA). Section VLA focuses on grading's intended effects, namely on the risk of foodborne illnesses. While we find no evidence of intended health benefits, Section VLB provides evidence of an unintended cost in resource allocation. A.Health Ourcomet The ultimate goal of restaurant grading is to reduce the risk of foodborne illness. The prevalence of foodborne illness from restaurant consumption, however, is extraordinarily difficult to measure.iBB Most instances of food poisoning do not result in formal complaints, news stories, or hospitalizadons. Moreover,the source of food poisoning may be difficult m trace.We here explore several indicators of food poisoning to assess the impact (if any) of the grading system on public health outcomes.aa'The indicators are imperfect,but if the health benefits are anywhere near as large as the zo% reduction in hospitalizations repotted for Las Angeles,we should nevertheless expect to detect some effect. 286.The Centem for Disease Control and Perventiou estimates that there are roughly 48 million cases of foodbome illness each year, but only 128,000 hospitalizations (less than o3%of cases). See CDC Rsdmates of Food6ome Mans in the United Sores, CeHrEns con DrsE CONTROL & PaB col (Feb. ton), li".//www.odc.gov/fwdborLiebxirdeVPDFs /FACFSHEET_A—FINDINGS_npdated4-13.pdf; see also Cus. for Disease Control & Prevention,Sarveillancefor Foadbom,Disease Owboos6—United Scares,soot,58 Moaamm &Mcaream Wray. Rep. 6o9, by (aoo9) ("(O]nly approximately half of dee [1,27o] reported outbreaks(of foodborne illness]in zoob had a confirmed etiology....');Paul S. Mead et al., Food-Related Mans and Death in the United Sones, 5 EMERGING INITcTTOUS Disnwsrs bot, 6o9 (1999) (discussing factors complicating die surveillance of foadbome illness,imluding underreporting). tet. We did not examine hospitalivdon data,which are available only via a protracted and costly ilfformadon request.HospiW izadons are an bnper!t mcasum of food poismdng incidence because oily a very small number of cases of food poisoning result io hospitalization,and diagnoses cannot be readily paced to restaurants. Sm amvene cited supra note 286. Constructing a credible mutual group To New York Cir,is not svaightforward. 643 THE VALE LAW JOURNAL 123:574 2012 Figure 14. CALL COMPLAINTS OVER TIME All an Co nplaiucs Allticnaorant ComPlain's foal PoisoOing n=9,856810 '13.0 it IRadantylnascas�Gad2age Smoking ViOhdon roreignObjeain Loud In-6.136 n-J,64) n-1Ax; mrtoltardPaal�n.a Poal8 Spoiled s e0ad6 aionh g{3t�crtcdN Kitrben/r Food con/ 1'oikt ru Fe n=820 n�T31 _� 11'469 _r 20 —`o 09 sono ami Eaeh pnml depicts coon.of 31,dl complain.on doe y-axes a01-1 mond.on the x-axes."rhe dashed,vnticgray lines indicxm du oof restaurant grading.The wp left panel presents all complain.. The second top my- panel pramts all resrautam complain.. The remaining panels present any restaurant complaints made over 450 times during the observadonperiod. First, we foctss on 311 call data. New York§ 311 phone line is a centralized information phone line for city agencies and services. During our observation period, it also served as the official route by which to report restaurant complaints, including food poisoning, to DOHMH. DOHMH itself has used complaints as a measure of performance. If there were substantial improvements in sanitation,we would expect 311 calls to drop with the onset of gading.'av But this is not the ase.Figure 14 plots time series from 2009 to 2011 :se. See The Mayor's Mmwgon cot liepen,PnGmin ny Fund 2012, CITY OF N.Y.3-4 (Feb.zona), h.p://—.nyegov/hM fops/dowdoads/pdUmmr/0212_mmr.pdf (citing decreases in child-we and pest-control complaints as evide cc ofdepamnental performanec). aay. Of course, grading might also heighten consumer awareness of sanitation conditions or DOHMH's exismuce,which might increase 311 calls.In is possible that the constant rate of 6" FUDGING THE NUDGE Of types of 311 calls, with the gray vertical fine indicating the onset of grading. The top left panel plots all 4.8 million 311 calls, the volume of which is stable during these years. The second panel in the top row plots the time series for all restaurant complaints, over z3,000 total. Each subsequent panel plots a type of complaint that was lodged at[east five hundred times and is related to restaurant sanitation (e.g., food poisoning, rodents, bare-hand contact with food). The grading intervention has little association with the volume of any of these calls. Second, we examine Google search trends,which have been documented to reflect general public health outcomes both temporally and geographically.'"If grading has a sharp effect,it should manifest itself in the search volume for food poisoning. For most cases, consumers may be more likely to search online for home remedies than to check themselves into a hospital. Moreover, such search data allow us to leverage both temporal and geographic differences(a difference- in-differences design). The left panel of Figure 15 demonstrates that search activity can reflect foodborne outbreaks. Beginning in July zott, for example, there was an outbreak of listeria tied to contaminated cantaloupes.The search for "listeria" spiked in late mii and did so, as we would expect, more sharply in Colorado—the origin of the contaminated cantaloupes—than in New York. (Cantaloupes are of course shipped across state fines, so we would not expect search activity in New York to remain entirely unaffected) The right panel plots search activity for "food poisoning" in New York in dark gray and neighboring metropolitan areas without grading systems (Albany-Schenectady-Troy, Rochester, and New Jersey) in light gray. The curves are indistinguishable both before and after the implementation of grading Vvalue= o.o98 for,if anything,a positive effect),""providing little evidence of a benefit in public health outcomes. calls is thereby consistent with a sharp increase in sanitary,conditions. Grading may also rause consumers m substitute,for example from'B'-grade to W-grade restaurants,which might relatively increase complaints in high-grade establishments and decrease complaints in low-grade establishments. 2ya.SerJeremy Ginsbug et al.,Detecting Influenn Epidemic,Using Smreh Eagim Query Dara,457 NATuNs 1012, 1012 (2009) ("Because die relative frequency of certain queries is highly correlated with the percentage ofphysician visits To which a patient presents with irdluerw- like sympmms,we can accurately estimace the careen[level of weakly influenza activity in each region of the United Sates,with a reporting lag of about one day.");Camille Pelat et al., Fatter to the Editor, More Direaso Track" by Using G.gk Trends, u5 EMERGING INeeerlous DistmEs 1327,1328(smog) ("[F]or each of;in&ctiouu diseases, 1 well-chosen query was sufficient m provide time series of searches highly correlated with incidence."); An Seiher et al.,The Utility of"Gwgk Tratuh"for Epidemiological Rararch:Lyme Disease as an 1uampk,4 GEDSPATW.HEALTH 135.135(2010)("Google Trends...approxhmme[s]certain trends previously identified in the epidemiology ofLymc disease.'). This p-value is fmm a standard diQemnce-in-dam ences It.,squares mgrcanm model with search wlume as the dependent variable and region fed-effe¢s,a post-Judy tom indicator, 645 THE YALE LAW JOURNAL Lia:574 Tntr Figure IS. INTERNET SEARCH ACTIVITY �LISTEMAT FOOD POISONING" Neighbors o D o - ColomM :5 EO � w 0 kow�low New York New York City oi 0 1008 1010 o 2008 x010 Time Time Th .. represent months and they-ixes represem volumq srandacdized so that too -pa—lies the miximum and o represents die minimum volume. The left pv111 presents time trends far Naw York(light gray)and Colorado(dark gray)for the word "listeria" on demnnion. that search activity on, emaningfidly reflect public health emblems.The spike in lam zou mrrespoods to the listeria outbreaks smmming fiom Colorado s'aLLtaloupes.Tlm right pvid presents seardt activity for-food poisoning"in New York City (dark gray) and neighboring maropoliwi areas fiom Abmy- Schcneaady-Tmy,Rochester,and Ncw Jersey(light gray). In short, we find no evidence based on these indicators of positive health effects. Over rhe long run,such benefits may still Lnaceriahze,but the evidence does not corroborate DOHMH's own claims of the program's benefits in the first year,�" nor the rapid and large effects for Los Angeles.X93 Perhaps this is not surprising—after all, the grades themselves do not Convey meaningful information that would enable consLuners to choose between establishments based on the degree of health risk and an inmiscdon term for New York City and post-July aoto.The mea ient on dne lines, an annulate of the treatment allco,, is an increase ,n search volume of sot units, Plus Or minus eight at a 95%mnfidence level. isa. See Ehno urant Lets r Grading:The First Your,raps note 234,at z('Grading has convibmed to sioticam improvements in mstaumvs'food safely practir s.'). zys. Sm Jin&Leslie.vryra nom M.at 4z6(detecting sharp public health benefits within one y<ar of letter grading in Los Angeles). 646 FUDGING THE NUDGE B. Perverse Resource Allocation While the evidence of the impact on health outcomes is weak, it turns out that grading has one strong effect, namely on the internal allocation of agency resources.One of the primary changes in the inspection system in 2010 was the introduction of reinspection solely for grading purposes. Most of the reinspection center around the 'A' threshold of 14 points (for example, the modal reinspection scores are just under 14 points in the reinpection column in Figure Io). Prior to grading, the primary on-site visits to restaurants following an initial inspection were compliance inspections. These focused, by law,on restaurants scoring worse than the'C' threshold of 28 points.X9'While compliance inspections still formally exist for restaurants scoring above 28 points,'9' we find startling evidence that grading displaced agency resources away from compliance inspection (generally at worse-scoring restaurants) to reinspection (generally at better-scoring restaurants).iq' The left panel of Figure 16 plots the proportion of all inspection that are compliance inspections over time. Each dot represents the proportion in one month, weighted by the total number of inspections in that month, with 95%confidence intervals. (The light gray curves plot the 95%confidence interval from a generalized additive model.) Prior to grading, 8-15% of all inspections were compliance inspections. After grading, that proportion dropped sharply to less than 5% of all inspections. The right panel plots the proportion of reinpection, which increased sharply to adjudicate grade disputes. Unlike compliance inspections, however, 58% of these reinpection are of restaurants initially in the 'B' range.�9J Grading thereby causes inspection resources ro be shifted toward higher-scoring 'B'-range restaurants and away from lower-scoring'C-range restaurants.As far as we're aware,no proponent or opponent of restaurant grading has articulated this concern. In a world where most health departments fall short of the FDA recommendation of a 294 132 CITY REC.884(Feb,18,zoog). ays. 137 CITY Rae.1607 Unite Is.zo I.). a,G The public health evidence on die relationship between saturation scores and health risks is inconclusive.Compare Irwin it al.,supra note 128(finding saturation scores to be predictive of foodborne illness), with Jones et al., snpm note 115, at 688 (finding am statistically significant association berveen saturation scores and foodborne illness). Given this conclusiveness,there is no obvious way To assess die diff nnrial risk from an W m a'e'w a'C'rescauranr.In die Irwin study,du point at which risk for foodborne illness increases is an"tweemsfuroty" inspection,suggesting dui there may be a threshold at womo-scoring ranges of the score. a93 This proportion is of all reirespections fo8owing an initial inspection for which rhe initial inspection resulted in a score of 14 or more points. 647 THE YALE LAW JOURNAL J¢2:574 aou minimum of two inspections per year (see last column of Table n),'va resource allocation matters. Figure s6. GRADING EFFECTS ON AGENCY RESOURCE ALLOCATION COMPLIANCE INaE.MONs nums'.enum. Gndiil no mldnnemnl O o o • a � IIS A. oJaooe s/2009 Jzoto J2m1 oJaoos Jaoo9 Ja ro thou Time T1014 The x-axis mp,me. months, and the y-axis represents the proportion of all inspections.Each dot represents the proportion of that month's inspections[liar were compliance inspections(left parcel)or reinspectimss (right panel),shed propOrourhally ro the number of inspections,with a vertical bar indicating a 95%confidence interval. The vertid guy line repruenrsde onset of ma corn grading,after which«sources shift sharply from compliance inspections (at the failure threshold of 28 points) ro reinspection (a grading thresholds). The light gray curves represenr dm predicted (pmm,vise) 9,% car5dena Interval from a genecalied additive model allowing for smoothened trends before and after grading,with a sharp break for July 2y,sum- Several caveats should be mentioned here. First, recall that New York also sought to hire twenty new inspectors with the onset of grading. The combined etTea of adding a reinspection system and[weary new inspectors may be that the total number of inspections at initially 'C'-range restaurants may not change as sharply as the left panel of Figure L6 suggesm.'v" However, we should distinguish between reforms that impose grayling—virtually always accompanied by a reinspection system—and those that step up enforcement resources.Adding 2 9 8.t).S.For.I ruts H SERV.,FOOD&DRUG A mw..supra nom ma,ar 204. 299.Thus inference is complicated because of dse mormansmnry of scores. Ideally, we wonid calculate dre average frequency at which a'C'restaurant is cospecred under die pre-zom and -zom tems.But die populatimrs ofpre-2010 and post-2010'C'-range restaurants are posh sot pafearysly compscablq because(q mstamarrs dim received a 4 On a post-2010 initial inspection and a B' upon re.numntion world have been a 'C'-range restaurarm before Binding,and(a)some pre-2coa.T-range remanants might have received'C'-range scares had they been reinspected. 648 FUDGING THE NUDGE grading alone draws inspection resources from somewhere else. Second, recall that the zom reforms also accelerated the inspection cycle for low-scoring restaurants.One might argue that there is simply less of a need for compliance inspections post-zoio because reinspection take the place of the first compliance inspections, and the next initial inspection comes sooner for some restaurants?°° Before zoio, however, DOHMH already had in place as "Accelerated Inspection Program" that increased the frequency of inspections for high-risk restaurants."' Unfortunately, the program policy is not spelled out in sufficient detail to understand the exact impact of the 2010 reform. Moreover, compliance inspections pre-zoto were conducted fifteen to forry- five days after a failed initial inspection and repeated every fifteen to forty-five days until the restaurant came into compliance.'°' Post-203o, reinspection occur within roughly thirty days of the initial inspection,—and the next initial inspection does not occur until ninety to r90 days later, even for the highest- risk restaurants scoring z8 or more points on any inspection.'*`Reinspection alone thereby cannot fully compensate for compliance inspections. Third,DOHMH also deploys inspection resources solely to monitor proper posting of grades.'°`These resources,again,must be drawn from somewhere. Last, the redistributive shift is further complicated by the fact that the underlying inspection scores are quite noisy. If the difference between an initially 'B'-range and 'C'-range restaurant is not meaningful, then the reallocation of enforcement resources may not [natter either. (Though recall from Figure to that shutdown of restaurants certainly take place at worse- scoring ranges;distinctions at the'B'-threshold may be much less meaningful than distinctions at higher ranges.) Tatung the inspection system on its own term,however,grading focuses resources on generally cleaner restaurants. Although targeted transparency is often billed as a cheap regulatory tool, grading in fact has concrete costs.In New York's case,two of every three initial inspections require a plenary reinspection,'oe and the majority of these 3m Along similar lines,pediaps the penalty of deceased customers is so mndr sharper upon recipt ofa'C'grade chat d3ere is no need in conduct compliance inspections. 301. aoog AumT,supra nore zoo,at 5 At n.a. 3w. Sm id at 9. 303. Sm Restaurant Letter Grading:Thu Fist Ymr,mpm note am,at 3. 304.Sec 97 CITY RED.1607(June 1S.atuo). 305. See Romumnr Liver Grading: The First You,supra note 034,at S("hrspecors also conduct targeted,ammounced inspadons for cud posting compliance at mraurants required to be posting B or C cords.'). 3o6.td.at 3. 649 THE YALE LAW JOURNAL 121:574 aola reinspection are of initially W-range restaurants. In retrospect, the resource shift may appear obvious. To provide a kind of safety valve to restaurants, grading was introduced in tandem with reinspection. The political economy of grade reform may well explain the design of reinspection,but reform comes at a previously unrecognized institutional cost. Viewed in light of these costs, the discontinuity of scores around the 'A' threshold may, counterintuitively, constitute a positive development.Inspectors may be compensating for the cost of grade resolution, assigning Ws to borderline restaurants so as not to waste inspection resources on establishments that, at least subjectively, do not pose grave public health threats. One of DOHMH's responses to the 2009 audit was that it would begin"in July zoro to post letter grades at all restaurants and further increase inspections of poorer performing restaurants."ao7 The conjunctive here actually masks a tradeoff. VII. IMPLICATIONS A.Design Matters How do we design a better grading system? We now articulate policy implications of our study that may apply to grading jurisdiction specifically and inspection systems generally. First,our study underscores the need for transparency about transparency. The availability of rich inspection microdata empowers information intermediaries to rigorously examine how well food safety programs function and to convey that information more persuasively to conumers.jo8 As Sam Issacharoff argues, "What is needed is a regulatory regime that would promote a market for intermediaries."" The Obama Administration's emphasis on microdam disclosure potentially facilitates such intermediation?- Indeed, the brunt of this Article an be considered a form of information intermediation that sheds light on restaurant grades. New York—one of only several major metropolitan areas that makes microdata readily available (see Table i)—is a Fit. am,AUDIT,man,one ao4,at 6(emphasis added)(quoting DOHMH officials). toe. For example, seorccard.org uses toxiary reports to convey health haaads more mevungfully m consumers. See SCOREWtD: THE pO,,mON INFORMATION SITE, hap://smrecard.goodgaide.mm(last visited Feb.a,sora). ,as. lssachaadl o.po,hose Ia,at 66:.Alli FUNG ET K.,raPm note 1;at tax-a6(dlsNasing the importance ofioformation intermediaries). era. Mcmoraadum from Sunstein,Julio nave v. 650 FUDGING THE NUDGE model jurisdiction in that sense. All jurisdictions should follow New York's lead and release full health-inspection data in machine-readable form. The disclosure should be comprehensive, including inspector identification codes, specific violations and point scores, types of violations, and data from restaurants that no longer exist.Even New York falls short of this goal,making it much more difficult to comprehensively assess its grading system. The benefits of wholesale disclosure extend beyond policy evaluation. Wholesale disclosures empower intermediaries to deliver information to consumers in more direct and effective ways. Inspection microdata, for example, would enable Yelp, a website that aggregates information about ratings of local businesses reaching roughly 66 million unique visitors per month,"' to include health inspection data in its restaurant characteristics. Similarly, the website Scorecard compiles data from over four hundred government and scientific websites to provide environmental information about localities.3" Disclosure of real property records by some and local government agencies empowers intermediaries like Zillow, a website that uses fine-grained information on too million homes,"'to deliver simplified, useful information, such as local home-value trends that are based on housing-price models, directly to home buyers. Smart phones permit dissemination to the immediate time and place of decisionmaking. Second, inspection criteria should be simplified to reduce variability across inspectors. The same behavioral insight of simplifying information for consumption should also apply to information generation. New York for example, could adopt a scoring worksheet closer to San Diego's, which would likely increase consistency across inspections. Ideally, agencies would conduct experiments to choose violation items and to determine the optimal level of inspection worksheet comple oty.s'4 A complementary approach would be to conduct more frequent, but shorter, inspections of a random subset of violations (weighted by risk).Such an approach might enable more objective measurement because inspectors could focus on a smaller set of more easily measurable 3n. Sce About Us,Yate,hnp://www.yrlp.wm/alwut (last visited Apc 8,xora) ("Yelp had an average of approximately 66 million monthly unique visitors in Qq 2011."). 31z. See Mout Smrercard: Scorecard's Data Sources, SMREGVtD: 7 HE POLLUTION INFQ0. ION Sera,hap://scorecard.goadVi&.mWabour/mr/dam.hmil(Wa visited Apr.8,2012). 13. S. "I is Mo.?,Ztuow,http://..zi to =a wrp/Abouth= (last visited Apr. 8, xora). V4- For a3 inadvertent rxperiman and discission of how to do scale equating to bridge distinct forms,see Daniel E.Ho&Timothy H.Shapiro,Evaluating Course Evaluation:An Empirical Analysis of a Quasi-Exper(mme as the Smnford lnw Sdwol,2000-2007,58 J.I.EOet EDUS.388 (2008). 651 THE YALE LAW JOURNAL Lxx:574 zona violations (e.g., food temperature of three randomly chosen items) and restaurateurs would have little time to clean up during the inspection.Removing inspector discretion by design (i.e., by random selection of objectively measurable indicators) may greatly improve the accuracy of inspection scores. Modem survey measurement relies on the same principle: random sampling of respondents removes surveyors'discretion to choose respondents.313 overly complex criteria appear to undermine inspections in other regulatory fields. As John and Valerie Braithwaite convincingly demonstrate, the complexity and specificity of criteria plague the consistency of nursing home inspections.311 Similarly, inspections by the Mine Safety and Health Administration (MSHA) and the Nuclear Regulatory Commission (NAC), which have no formalized score sheets"' despite a large number of possible violatiorm,na are subject to sharp criticisms of incoasisteney.319 The Bmithwaites argue that simplification in particular promotes consistency by 313. Quota sampling, for example, in which surveyors those respondents within subsets of covariates(age,race,gender),infamously introduced substantial bias.See DAvm FREEDMAN, ROBERT PssAM&ROGER Puuvrs,STATUMCS 33]-39(4th ed.zooT). 3,E. See Joint Braithwaite&Valerie Braithwaite,The Politic ofLqulb.:loale, ibn Nursing-Home Regulation,4 SO,&LEGAL STUD. 307,3e, ( 995) C ratings of the qu2hry ofcue in nursing homes are possible when professional raters use a limited number of criteria;bill when racer use the large number of specific American cegulatwtss as their criteria,reliability is loco'). 9 P gmmf° Dry Sromgrc of Spent aim for 3+3. See NRC ag radon Mamma Chapter 161a'Lis inion Pro Fuel Al g,NUdnu Spell, REG Fuel Sromge Ir-talfatiorr and for to CFA bpaadduya s ra'cr� dors Packagings,NOCLEAR AEG.COMMnaafro 9-10(Mar.9,zawn,hap S Al � god/ /ML11o;/MLuo39og15.pdf;E-mail from Rodney M.Brown,Miste Safety Al Health Admin.. to Mridula Raman(Mar.z6,2012)(on file with author). 3,E. For mining safety,am 30 GF.A§$1-104(2012).The regtds th ationerem'aim to cegohne all aspects of miner's[sic]safety and health."Jay Lzpat&James P.Notter,Impxting the Mine bespo,la, Why the Discretionary Function Exaptom Dau Noe Bar Cooemmem Liability for Negligau Mine Inspecowns, a3 HOESTM LAB. &Emv.L.J. 413,46 (2006). For tegtdatime concerning spent unclear file[storage,see 10 C.F.R.5 n(2ou). seeOffice aftnem=pea9rcen..auditRgNn:Anis afNRC;Oweig6-J, (Mara<trspa nol), Storage Inuallanum Safary, Nu¢ REG. CoAtmLnioN 6a5 (Mar. 19. hon), http://www.tuc.gov/reading-m✓doc-collectiolss/resp-geq/zou/oig-u-a-az.pdf(NAC): Ausun Hoffman, South Dakow i Mining Controversy, KELOLAND.0 Y4 (May 13. 2011, 9:58 PM), h".//www.kelolmd.mm/news/eymnd iolmd/Ne Dleeud646403.cf�uu.NEB.CONCRETE Iavmnkm Express Comm and Seek AnasoY R%www,nebrwm� ro astt ]ONCAETE &AGGREGATEEAsS NNEW l-(Dec.am'),hirp:// gg W genic nmz mcrs /Deccmbetzou/Denmews_final%zolow%zores.pdf (MSHA); John Thunn. Mires Impetlom Harm SD Bmivuc,Job Crmdoq]oHN TH NE:U.S.SENATOR—S.D.(May 13,2o1n),http://www.rhnrne.senare.gov/Pnblir/index.cfiltholl/S<tuorssisteno-anne-bupections -harm-sd-busiiusses-job-creation(MSHA). 6¢ FUDGING THE NUDGE fostering deliberation and a form of peer review among inspectots.3- Onr findings corroborate that simplification on the information-supply side may improve inspections in other regulatory areas. While our evidence suggests that reforms would reduce the impact of the inspector lottery, the major remaining limitation lies in inspection resources. Without sufficient supervision and training of inspectors,"' it may not be possible to achieve satisfactory uniformity across inspections. From that perspective, the more difficult policy decision may be whether to increase the budgets and salaries of health departments. Third, inspections should take place at truly random intervals to eliminate short-term changes taken solely in anticipation of the inspection."" A pernicious feature of existing regimes is the relative predictability of when inspections will occur. In San Diego and Los Angeles, restaurants can pay for a next-day reinspection. In New York, the July nolo reforms spelled out in concrete terms when to expect inspectors—seven days to roughly a month for reinspection, and ninety to 15o days for the next initial inspection for restaurants receiving 28 or more points.3�1 Such certainty enables restaurateurs to devote resources to a temporary cleanup in advance of the inspection. Greater randomness would make such strategic cleanups far more difficult. Increasing the randomness in timing of inspections takes real political will,but making inspection scoring more consistent may reduce restaurant hostility toward grades,making such reform more feasible. Sao. SmBraithwaite&Braidrwaite,sapnnom316,at319-u. in. Sce Mlchclle Com rdrio et al.,Effort ofa Manage Training Program on Sani ary Condition in i somans, 13 PUB. FIMTH REP. 353 (1998) (finding dnat manager training and certification programs may lead co better sanitary conditions in restaurants). 3n. Ofcourse,there are diff nt forms of randomness.A simple rube would be shut an impector could appear for the next inspection in a random interval between one and 365 days after on inspection has oramcd.Stratified randomization to account for die risk of an eseibhslmant is also possible.For example,an inspector could appear in a random interval hetween one and 182 days for"high-risk"resraurams(however defined)and 183 mid 365 days for"low- risk restaurants" (however defined). For some discussion of randomization to; reduce behavioral biases in the el omrs context, see generally Dand E. Ho & Kosuke Imai, Enimating Caural Effa ofBauot Orderfram a Revdwoused Natural Experiment]Re Calfmw Alphabet Lottery. 1978-2002, I2 PUB. OPINION Q-216 (mo8), which shows that cognutive limitations that brad wmre to be aff ted by ballot order can be overcome by randomiradon and coomion ofballot order across districts;and Daniel B.Ho Be Kosuke Imai,Random cation Iryinne,with Natural EVaimenn:An Analysis of Banot Effe in rhe 2003 California Recall Election,011 J.Ata.STAT.Ass'N 888(amm),which shows die same. Tu Sw 137 Ctry REC. 160 (June 15, 2010); How We Scare and Grade, supra nom 2q3 ("An inspector goes back to the restaurant unannounced,typically within a month...."(emphasis added)). 653 THE YALE LAW JOURNAL taa:574 rDra Fourth, to battle grade inflation, jurisdictions like San Diego should consider changing the thresholds for letter grades to generate meaningful distinctions. For instance, if San Diego employed a threshold of 95 points to receive an 'A,' consumers would receive more information about the relative risk of establishments. At minimum, the overall proportions of restaurants receiving each grade should be disclosed on the grade placard. Last, health departments (or information intermediaries armed with more comprehensive data) should apply well-known statistical adjustments for differences across inspectors and inspections."' The intuition behind such models is that good stores by tough inspectors are more meaningful than good scores by easy inspectors. Statistical models can adjust for inter-inspector differences so that the numerical score is comparable across restaurants, regardless of what the grade threshold may be. (Insights from such models could also be applied to adjust for the time of the day.) Moreover, any discosure to consumers should convey uncertainty in the scores.rn For example, one simple proposal would be to disclose the (model-based) probability that a restaurant would receive an`A' if inspected on a future day. Such adjustments would appropriately tailor the strength of the disclosure to the consumer by the uncertainty in distinguishing sanitation levels of restaurants. New York's grades aim to cure an information deficit but,if anything, may overcompensate by creating a false sense of certainty. B.Retargeting Transparency Beyond these spec design elements,this Article raises profound questions about mandated disclosure and targeted transparency.First,given that the poster child of targeted transparency is it"susceptible to ineffective implementation, this study raises questions about the design of disclosure policies far beyond food safety. It calls into question the design, implementation, and administration of rap. See, e.g., ExPIANATORY ITEM RESPONSE MODELS:A G£NER.W2ED LIN£Ut AND NONLINEA0. APPROACH(Paul De Boeck&Mark Wilson eds.,ao)4);Jf Michael Peass&Arthur Spirling, Swung the Critics: u.ioing the Inaeru Danmsiom of Movie Guaieum unth an In.R.pome Approaeh, 105 J.AM.STAT.ASSN 71 (aoro). Applying Iran response theory, for example, inter-rater adpaurnenrs maid be applied m acmant for differenu across inspamrs,while differential item functioning could be applied in nst for dre timing of dm inspection. ras. See Daniel E. Ho&Kevin M.Q itm,Improving dw lonarmanum and Inunpr.Arion of Online Raaf g,Doan uwh Model-Blued Figur.,6a AN.STATISTICIAN 11,a1(aoo8)(observing Liar mrrent practice in milinc ratings fails to inmrporem searsucal uncerrunry). 654 FUDGING THE NUDGE disclosure in a myriad of regulatory areas.3a6 Second, while behaviorally informed regulation is an extraordinarily promising approach, the contextual nature of behavioral effects also makes it difficult to extend findings from one arena to the next.Nudges are contextually dependent. A yellow 'C' grade, for example, may have quite different effects from a red 'C.' New York already had a means of publicly indicating positive sanitation results prior to July 2010—the Golden Apple—but one that apparently did not function effectively. What our findings underscore, then, is the inaeasingly recognized need to evaluate empirically the efficacy of such design elements, with field experimentation being the most credible assessment tool.327 Fortunately, the changing evidentiary base of government, combined with the increasing availability of rich microdata about and from administrative agencies, facilitates the systematic assessment, understanding, and, ultimately, improvement of the regulatory state in ways previously unimaginable j" Third, nudges cannot compensate for underlying problems in regulatory design. Slapping a grade onto a score from a faulty inspection system provides the imprimatur of transparency,without a public health basis.If the simplified grade or score is merely a proxy (that is, if it reflects but does not directly measure the concept of interest, namely the risk of foodborne illness), it tan be aa6. Cf. Ben-Shahan & Schneider, supra nose ra, a 743 (arguing that mandated disd0sum generally fails across substantive areas, but that restaus nt letter grading is one effea{ve example). au. See Michad Abramowi¢, Ian Ayres Be Yaw Ustokin,Randomizing Law, 159 U.P&L REv. 929 (2012); Gary Icing « al., A "Politically Robust"Experimental Donn for Public Polity Evaluation, with Appliesion w the Me icon Univenal Health Imurame Program, 26 J.Pot'v ANAEY615&MGMT.479(-07);Christine Jells,Review;of Draft 2011 Report ta Congew on the Ben fin and Cons of Feleral Regulmions and Unfunded Mandmes on Scare,Latin, and Tubal Emitter,OFFICE OF MGMT.&BUDGET(2au),hap://www.whitchonse.gov/si¢s/defaldt/files /omb/mkrcg/2ou_cb/commenrs�olls.pdf (calling foc expenmennl assessman of regtdaturypolicydtanges). 32s. Sar,eg.,FVNG ET. ,supra note q a 170-82; David Bonier,The Promise and Peril of Bg Data, ASPEN INsr. (2010), li".//www.mpenhudmm.org/sins/default/files/wnwnt /doa/pubs/Cln:_Promise_and_Peril_of ,Big_Dan.pdf;Dow Mining:Federal Effort,Cover a Wide Rage of Usa, U.S. GEN. Acer. OFFICE (2004), htep://www.gao.gov /new.items/do4548.pdF Daniel C.Esry Be Reece Rushing,Governing by rhe Numbers: The Pmmue of Dots-Driven P ixymaldng in the Information Age,CTR.FOR AM.Pur a uss(2007), litcp://xww.americanpmgress.org/wp-commnr/uploads/issues/too//o4/pdf/dara_driven -policy_repon.pdf; Daniel E. He, Big Data, Small Tax Gap? Detecting Cash-Only Tax Evasion in Madutan Restaurants (Nov. 8, lou) (unpublished paper) (on file with author);Gary King,Albert J.Weeherhead III Univ.Professor,Harvard Univ.,Hornolu in Political Science Talk at the Harvard University Govemmem Department. The Social Science Data Revolution(Mar.30,ea,). 655 THE VALE LAW JOURNAL Jai:574 aUti strategically gamed by restaurants and inspectors,thereby losing validity.3' Fourth, the broader desirability of grading (and nudging) depends on a normative theory of the regulatory regime. Is the purpose of such systems, for example, to identify sanitation outliers? In that respect, San Diego's system actually performs far better than New York's: a B' is truly informative and heightens the expected penalty of noncompliance. Or is the purpose of the system to incentivize restaurants to improve across the board?In that case, we might favor more grade discrimination between restaurants, as in New York. Given fixed resources, however, the latter comes at a Considerable cost—a reinspection system for grade resolution, which is part of every mandatory grading jurisdiction we have examined. Inst, our findings also point to the political economy constraints of regulation.Disclosures,like bureaucracies,are"not designed to be effective.11330 The lurking political economy explanation for grade reform may be that it simultaneously allows an administration to visibly and publicly claim credit for transparency, while providing sufficient assurances for the regulated industry to contain grading's impact in practice."' CONCLUSION Targeted transparency remains one of the most promising regulatory approaches of this generation. As the rare instance in which disclosure cats broadly affect behavior, it has the possibility to transform mandated disclosure into a genuine tool for empowering cognitively constrained consumers. This Article has shown, however, that even the perceived paragon of targeted transparency can be seriously flawed in implementation. Our examination of over 700,000 inspections in San Diego, New York, and eight other jurisdictions shows that grades can be uninformative and costly. Targeted transparency cannot solve or avoid the core issues of administrative Sas. This is known by some as"Goodhart s law.'Su generally Sanjai Bhagar,Brim Bolton& gobeaa Romano,The Promise and Peal of Corp om,Garoumre Indices,308 COLUM.L REV. 1803(1000)(analyzing the difficulties of summarizing coria me governance practice 1n one index);Esty at Rushing,supra note 328,at 38-39 (discussing Trow data-driven governance can improve decisioncraking but noting the risk that scarecords can misdirect attention and inceudves"). 310. Terry M.M.,The Polities ofBureaunanc Smxzunt,in C.w THE GOVERNMENTGOVERN?267, Z67 Qohn E. Chubb&Paul E.Peterson eds., 1989) ("Alnenan public bureaucracy,s not designed to be effective.'). 331. S. FONG ET N.., supra note Ja, at m6-x6 (discussing the sustabubdity of targeted or ,uparency given the political context). 656 FUDGING THE NUDGE law—the institutional design of inspection agencies, the development of administrable riles and standards, and the accountability and oversight of expert agents. Without these elements in place, health inspections cannot generate meaningful information, and targeted transparency risks turning into a facile mantra of regulatory refocm.J11 To quote the"nudger in chief': "[D)isdosure may greatly alarm people. .. without giving them any useful information at all."313 332. See Jerry L Mmhaw,Reinventing Govemmem and Regulatory Refo.m: Studio in the Neglect and Abuse ofAdminueranve fou,57 U.PITT.L.REV.405(1996)(discussing die deep tensioim between regalamry reform of on and principles ofadininistradae law). 333. Cass R.SUNSTEIN,LAw50FFFAR:BETONDTHEPRECAUTIONARTPRINCIPLE 123(ZOOS). 657 THE YALE LAW JOURNAL 122:S74 2012 APPENDIX Appendix A documents integrity issues that affect the DOHMH data. Appendix B spells out the details of the classification algorithm that uses our substantive knowledge of the DOHMH inspections system to classify inspection types. In a cross-validation test, it classified 97% of inspections correctly, with the 3% classification error largely attributable to underlying errors in the DOHMH data. Appendix C shows that types of violations are comparable before and after grading and that the consistency findings for New York remain the same when examining other types of inspections,adjusting for administrative hearings,and replicating the analysis exclusively from DOHMH website data. Appendix D shows that evidence from eight other jurisdictions confirms our findings above.Appendix E provides sources used to compile the information about jurisdictional differences in health inspections and sanitation grading in Table 1 and Table 2. A.DOHMH Data Integrity This Appendix reports in more detail the data integrity issues we discovered to affect DOHMH data. Table 4 summarizes major issues and also provides, where possible, the number of inspections affected. Although these inconsistencies might ultimately be explained—for example, by unobserved score changes after administrative hearings or unobserved changes in database conventions—the data that DOHMH has made available do not allow us to do so.We divide our discussion into errors that ran be assessed(I)by using solely the December 2011 version of the dataset ("December Version"); (2) by comparing the December Version and the DOHMH website; and (3) by comparing how the same inspection was represented over six versions of the dataset from January 2010, July lou, August 2011, October 2011, December 2011,and January 2012.Each of these versions covers inspections starting from at least 2007. I. December Version More than 6,00o inspections appear to violate DOHMH's inspection and grading procedures. Most commonly, an inspection has a grade that is inconsistent with its score (Table 4, row A). For example, the October 5, 1011, inspection at Ohiyo i-Cafe has a score of 12, but a 'B' grade. Over 550 inspections assign a grade but no score(Table 4, row D).The score and action code may also diverge. In 315 instances, an action code of'B,' for example, which in principle indicates that no violations were cited, accompanies a 658 FUDGING THE NUDGE positive score (Table 4, row G). Several inspections have an action code of'8,' which is not, to our knowledge, a valid code (Table 4, row 1). One possible explanation for such invalid action codes is transcription and data entry errors, which the 2oo9 Comptroller's audit also found to be significant." We also detect over i,600 inspections for which an initial inspection with a score of 13 or lower is followed by a reinspection (Table 4,row B)."' Table 4 DOHMH DATA INTEGRITY ISSUES A Gradeinronsou,mwirhswre 4.363 C. Inspection guide changes more than[wire over datasets 1 meq, ygrq 51 MWIM—W—P-00"PAW E. Score changes more di,m name owr,damsers SP F `.1`Gn q erDd$nSes n' rysSJr?na{" v' 368 G. Posiuvcscorebut'B'actionrode PS I. Action nide of's' is K. Websitememneedbutdanst"careho ? Dau integrity issues mumentered in an audit of six versions of the DOHMH dataset (bloom January 2010,July ami,August 2011,Ocrober aou,De¢mber-a0u,and January 2oia) and mdhm websire data. Although dsusre small relative inthe total amber of inspections, they suggest that DOHMH has not properly designed the database. The second column provides an estimate of die number of affected tutus, Mitre a '?" indicates that m estimate is not possible without direct across to DOHMH's underlying database. s. December Version and DOHMH Sete Even more concerning is that the December Version exhibits numerous discrepancies from the DOHMH website, despite the fact that we checked the 334Ser2oogAUDIT,supranote2o4,at, srs To estimate the frequency,we port die number of inspections with an action code of'P' (indicating that a reinspection was scheduled)and a score below 14. 659 THE YALE LAW JOURNAL 1zt:574 ¢o1n website against a dataset downloaded less than one month earlier. Unscored inspections on the website sometimes have a score of o in the December Version (Table 4, row K). Scored inspections that received o violation points on the website have missing scores in the December Version. The numerical scores con also diverge. The December 23, 201o, reinspection at Jenni Coffee Shop Corp.,for instance,has a score of 22 on the website but a score of o in the December Version. DOHMH also does not clearly distinguish between inspections. On rare occasions, restaurants are inspected more than once on the same day, but neither the website nor the December Version can distinguish between multiple inspections, and multiple (false) entries for the same inspection. On the website, for instance, Mantus Restaurant is listed as having two inspections on May 5, 2oto, both with a score of 25 and five violations; the December Version also records two inspections on that day,one with 23 points and five violations and one with 25 points and no violations. As another example, each of the violations at Imperial Bakery's July 5, 2011, inspection is listed twice in the December Version. Outright duplication, however, appears to be raze. In thirty-eight instances, a restaurant appears to be cited twice for the same violation at a single inspection. Finally, in several instances, a restaurant is listed on the website as having received two consecutive reinspection (Table 4, row L),which contradicts the New York City Rules336 piadina Restaurant, for example, received graded reinspection on September 7, tort, and September 28, Zola, with no other inspection occurring in between. 3. Comparison of Five Versions Although our analysis relies primarily on the December Version, we collected additional versions of the DOHMH data from January roto, July 2011, August 2011, October 2011, and January 2012. (Because DOHMH systematically deletes restaurants from the most recent releases of the dataset, these version are necessary to reconstruct a comprehensive version of the DOHMH data.) Comparing these different versions uncovers several thousand 336. Sm 137 Cnr Rue. 1607 (June 15. aoro) (defuc ng an "inspection cycle" as a series of inspections "ronsisting of at least an initial inspection and including, if niggeted by die initial or any subsequent inspections widain that cycle, a ¢inspection" and defirsng a "reinspecdon"as an"inspection conducted for the purpose of grading following receipt ofa score of 14 or more points on an initial inspection"(emphasis added));id. (providing that when there is an increased risk to public health ds Department may Nropectl] all establishment aid tear[]that inspection as the initial inspection in a new cycle"). 66o FUDGING THE NUDGE instances where a score changed after an administrative heating. However, we also discovered cases where there were more than two changes in the score or grade across different versions (Table 4, rows C, E, and F). The August 24, 2009, inspection at Vernisazh Restaurant, for example, had scores of 59 in January 2010,28 in July zou,it in August zou,and z8 again in December zou. An inspection sometimes also has a worse grade in a more recent version of the data, which, as far as we understand, cannot happen due to an administrative heating.The grade for the October 15,zou,inspection at Mi Colombia Bakery changes from an 'A' in the December Version to a 'C' in January x012, even though the score(8)is the same.An employee of Mi Colombia claimed that the grade was still pending. In sum, although New York is a model jurisdiction in making the inspection microdata available,the database exhibits an array of internal errors. While these are disturbing, their number is small relative to the size of the database, and therefore unlikely to explain our general findings. They do, however, provide an additional reason to question the reliability of grading in New York B. Classtf cationAlgorith n Our algorithm for classifying types of health inspections formalizes the inferences one would draw from the descriptions and information in the database and the Rules of the City of New York We focus on scored inspections in the post-grading period and use the following pieces of information: (t) the date sequence of inspections for a given restaurant, (z) "action codes"recorded by inspectors; (3) the grade assigned(if any)during an inspection; (4) the score assigned during an inspection and on prior inspections; and (5) the date an establishment enters the data. The DOHMH website distinguishes between graded, ungraded, and unscoved inspections. Graded inspections are either"[i]nitial inspections that result in an A" or"re- inspections that result in a grade of A, B or C."ss' Ungraded inspections are "scored but not graded" and include "[i]nitial inspections that result in more than 13 points," "inspections a new, not-yet-opened restaurants," and inspections"at restaurants the Department closed that are being considered for Sty Inspection type descrlptiom can be found on tine DOHMH inspection information page for any New York Sm. e.g., Retraction I yeubn Information, N.Y.C. Dse'T of HEALTH&Mewru.HYGIME,http.//"16-resnoraiidmpmdon.nyc.gov(last visited Sept.x, aota) (enter"toot NIGHTS CAFE' in the "Restaurant Name" field, click"List Results," dick on the first readq and move the mouse over the question mark nut to "Graded," "Ungraded;or"Unscored"in bring tip a description of that inspection type). 661 THE YALE LAW JOURNAL 12 x:574 zolx re-opening.""' The algorithm is somewhat complex, but can be sketched out, from the simplest to most complex decisions,as follows.First,we classify as a reopening inspection any inspection with the action code`O'or the current grade'P,'both of which denote that DOHMH reopened the establishment after a DOHMH- ordered shutdown.J3 Second, we classify as a reclosing inspection any inspection with the action code 'W,' indicating that DOHMH decided not to reopen an establishment that it ordered shut down. Third, to classify compliance inspections, we use the Rules of the City of New York, which provide that DOHMH "may . . . also conduct a compliance inspection after any inspection that results in a score of 28 points or more."31O DOHMH clarifies in a brochure that a "restaurant that receives 28 or more points on a re-inspection will receive a'compliance inspection' roughly 30 days after the re-inspection and that DOHMH "will continue to conduct compliance inspections roughly every 30 days until the restaurant scores under 28 points or is closed by the Department"3f'Because a new cycle is mandated to start "qo to 150 days after the final inspection of the cycle at an establishment that receives a score of 28 or more points on its initial inspection or reinspection,"31' the compliance inspection after sixty days effectively becomes an initial inspection.We therefore classify as a compliance inspection (3) any inspection following a graded inspection when (a) the restaurant received a score of 28 or above in one of its last two scored inspections, (b) the current inspection occurs within sixty days of the last inspection, and (c) the last inspection was not an initial inspection; and (1) any inspection following an ungraded inspection that is (a) not graded and does not result in a Shutdown (in which case it would be a reinspection), (b) does not follow a I eopening(or reclosing) inspection,and(c)occurs less than sixty days after the last inspection and the restaurant received a score of 28 or above in one of its last two scored inspections. 339. Ed. 339. Tlhe zeton Code'O'appears M indicate chat de resteurvtt reopened,while the current grade 'p'indiarcs aplaceholder for a grade during a reopening inspection. 340. 137 CITY MC 1607(June 15,2010). 341. Grading: What II Means,supra note r 2.at 3.This statement conflicts to some degree with New York's rules,which indicatethat compliance inspections may be performed after"any nsp iection'that results in a sore of 20 points or more.137 QTY AEC.1607(June 15.x010). Moreover, the DOHMH websi.and database d0 not record scores for the large bulk of compliance inspections, leading one in wonder how it is Out the Deparrment makes the judgment tut thesuvurant has scared below 20 points. tat 137 CITYAEC 1607 Untie g,mao). 66x FUDGING THE NUDGE Fourth, we classify as an initial inspection (1) the first inspection occurring after July 26, 2010 (as the Department used a sharp date cutoff to begin initial inspections for grading purposes); (2) an inspection that occurs after a reopening inspection;H3 (3)an inspection occurring after a graded inspection if the inspection is not a compliance inspection; and (4) an inspection occurring after an ungraded inspection that is (a) not graded and does not result in a shutdown, (b)does not follow a reopening(or reclosing) inspection,and(c) is not a compliance inspection. Fifth, we classify as a reinspection an inspection that occurs after an ungraded initial inspection and is graded or results in a shutdown. Lastly, an inspection occurring after an ungraded inspection that is neither a reinspection, nor a compliance inspection, nor occurs after a shutdown, is classified as an initial inspection following a pre-permit inspection if it is the first appearance of the restaurant in the dataseL For our purposes,we equate initial operational pre-permit inspections with initial inspections, as the subsequent inspection is typically termed a reinspection by DOHMH (e.g., the March 21, 2011, and April 27, 2011, inspections at Aunt Rosie's Coffee Shop and Diner). An initial nonoperational pre-permit inspection, on the other hand, counts as a pre- permit inspection for our purposes,as the subsequent inspection appears to be termed an initial inspection (e.g., the March 25, lou, and June 21, 2011, inspections at Bad Horse Pizza). For unscored inspections (used to examine the shift from compliance inspections to reinspection), the classification is simpler. Compliance inspections are unsecured inspections, unless DOHMH separately reports administrative violations when a scored inspection has been performed on the same day." Table 5 reports the results from a cross-validation sample (i.e., a random sample hand-coded from the DOHMH website). Importantly, the sample was randomly drawn and not wed to develop the algorithm. The classification accuracy is 97%.As best as we can tell, the small number of misclassifications stems from errors and inconsistencies in the DOHMH database that are not s4s. Id('M initial inspection mnunewing a new cycle shall be conducted within 6o to Do days of reopening for an establishment drat is audiori2ed by the Department to reopen following a Department closure that occurs on an initial inspection or re nspecoon of that anblishmenc'). 344. The DOHMH website describes un cored inspections as ones involving 'prohibiting smoking or the use of artificial crass fats or laws requiring certain chain restaurants to post calorie infmmauon.° Pmasmant Imimaion Information,Pyre note 337 (follovm g die same nsm¢tions). Even when a fill sarunry inspection u conducted, these admimstestive violations are separately noted in the unscored category. 663 THE YALE LAW JOURNAL raa'5]4 inti reducible and in fact appear to violate the Rules of the City of New York. Two misclassifications, for example, stem from back-to-back ungraded initial inspections. Three misclassifications stem from ungraded reinspection. In another instanre, an inspection is unscored on the DOHMH website, but receives a score of o in the data file. Table S. CROSS-VALIDATION }'O OF INSPECTION CLASSIFICATIONS TRUE TYPE INITIAL REINSPECT. PRE-PERM. REOPEN. RECWS- COMPL. 5 1 o o 3 1600,_,p? 0 0 0 0 ASO" - ° o o 0a�ir4.y �: 0 0 ° 0 0 0 0 0 A Cross-validation of the classification algorithm for scored inspections in dee post- grading period.Tlm rows represent the classification of5ol randomly sampled cues by our algoriclun.The r[mom represent the tare type,As hand-coded from the DOHMH site. The sample was a true cross-validation sample (i.e., not used to develop the algorirhm).Nmmy-seven percent Of inspections are mm,cdy classified in tris sample, As represented by the bottled diagonal numbers. C.Robustness t. Types of Violations Here we show that the few changes in violation codes over time do not affect our findings. To assess the potential impact of these changes, we remap violations from all pre-grading periods into their post-grading violation codes."' The left panel of Figure 17 plots the proportion of each type of 345. For example,we remap a rA violation to An r8A violation. See Bureau of Food Safety& Canty.Sanitation,Sdf-Impvxdon Workrhat or Food servicesmbltihmenu,ao.a Dcn'TdOf Hr,L,, & MENEAL HYGIENE 4 (io ). up:// Y .gov/hLAd/ lV da /pdf/dVself-inspection-worksti eL.pdf, Impaxiun scoring symno for Food service Faabluhmmu,supra more n4,ar app.a3B. 664 FUDGING THE NUDGE violation in the pre-grading period on the y-axis and in the post-grading period on the x-axis. Although there are some slight differences, the panel shows that the system has by and large remained stable in terms of the overall distribution of violations cited. The right pane[ plots the use of violation codes in initial inspections and in reinspection, showing that the Types of violations cited in these types of inspections are also comparable. Figure17. SUBSTANTIVE COMPARABIUNOF VIOU\TIONS PKE/POST POST-GRADING S co � ° o E 6 _� P • p •N s d \ o S • . 0 0. o u.00 o o.o; 0.a 0.00 o.04 0.08 Initial post-grading Initial The left panel plots die proportion of violation codes issued in iuidal inspections polo- grading on die x-axis and for the pre-grading period on die y-axis. The right panel ploy the proportion of violation codes for initial inpecdons on die x-axis and reinspection on the y-axis.The scoring changes from grading minimally affected die distribution of types of violadon found. Dots are plumed with gray transparency for visibility. z. Consistency in Other Types oflrupections We focus above on initial inspections across inspection cycles, as these are closest to random inspection.Several alternative explanations,however,might Geist. First, perhaps the initial inspection serves no purpose other than that akin to an audit lottery—i.e.,randomly generating a real inspection in the form of the reinpection.3 ' In that rase, we might want to focus on reinspection Sas. See McSwana et al., supra note 127, at W ("[M)any jurisdictions advocated using the inspection as a screening device to identify high-risk operations and those requiring closer scrutiny.'). 665 THE YALE LAW JOURNAL 111:574 2012 across cycles. The top-left panel of Figure 18 shows that the same lack of consistency persists when we focus on reinspection. Second, perhaps what [natters is whether the graded inspection (which could be an initial inspection or a reinspecrion) is informative about the subsequent graded inspection,even if reinspection are predictable from the restaurants perspective.The top-right panel of Figure 18 shows the same lack of consistency across graded inspections. Figuret8. LACK OF CONSISTENCY IN OTHER INSPECTIONS REINSPECTIONS GRADED INSPECTIONS o ,` 1 �1: 4 0 60 40 20 o to 40 xO o Cycler Cycle r SUBSEQUENT CYCLE PILE-GRADING _ O ..........._�... O E.: 7... ;�1. O 1� NP 0. m 6o 40 20 O 6o 40 20 0 Cycle z Cycle r 666 FUDGING THE NUDGE Each dot represents die scone a spaific restaorant received in die first inspection cycle on die z-axis and in die subsequent inspauon cycle on dey-axis.The first panel plots reinspection sonrex for restaurants diat received reinspection,in subsequent cycles; die second panel plots inspections resulting in a grade;die third panel plots the second and drird inspection cycles after grading began; and the folindi panel plots initial impauens in de pre-grading period.For visibility,obscrvarimu are randomly paired and ansored at sixty. Third,perhaps the lack of consistency is simply a short-term finding. Both inspectors and restaurants may require several cycles to team about the grading system,which would then induce some correlation across cycles. If true, this of course means that grades in the first few periods are essentially meaningless, which would contradict Jin and Leslie's finding that effects are realized within one year.'47 Moreover, the bottom-left and bottom-right panels of Figure 18 show that the lack of consistency persists across the second and third post- grading cycles and across cycles in the pre-grading period. Lastly, grading may have beneficial effects even if the inspection scores are uninformative. Grading may cause restaurateurs and consumers to be more conscientious of sanitation practices in a way not manifested in inspection scores. Alternatively, perhaps the real signal is not the grade that a restaurant posts,but whether or not it complies with the posting requirement.Customers could use the posting itself(regardless of its content) as a signal of restaurant quality. These conjecams; do not appear borne out by the findings of Section VIA To some extent they cannot be directly tested with the data at hand, but in any case they are also divorced from the concept of targeted transparency. 3.Administrative Hearings How do administrative hearings affect the inspection system? On the ones, hand, hearings [night be the root muse of inconsistencies. Differences across administrative law judges,for example,might create or exacerbate inconsistencies, in which case we would falsely attribute inconsistencies to inspectors. On the other hand, the hearing process might regularize outcomes. Administrative law judges,who, after all, observe inspections from multiple inspectors, might grant relief whenever an inspection appears to deviate from overall patterns. If so, the underlying inspection scores should cx:hibit even less consistency than we document above,as the scores in our data are largely post-heating scores. Unfortunately, New York does not disclose the results of hearings. From the December 2011 data,we do not directly observe whether a hearing occurred (or whether the notice of violation was settled before a hearing), or whether ay. See Jin&Uslie,supra nom yq,at qa6. 66 THE YALE LAW JOURNAL hzx1574 2012 the score in the dataset represents a reduced score from a hearing favorable to the restaurant or the score assigned by an inspector. To solve this,we compare multiple versions of the dataset(from January nolo to January xo 3a),which we privately obtained. Each version contains the then-current scores, a subset of which would be pending a hearing. If a hearing reduced the score, a subsequent version of the dataset would contain that reduced score.JOs As a result, for a subset of over sic thousand inspections (roughly 5% of our inspections)we can separately identify the score assigned by an inspector and the reduced score assigned by the administrative law judge. Although we cannot infer the overall success rate of appeals(our data provide a lower bound),we an examine whether harings reduce or exacerbate inconsistencies.309 Sae. It is possible tut some of the scoring changes stem from dara-entry errors dun are cocecmd over time. DOHMH provides no documentation for such dhanges. Because sconng reductions in the post-grading period stem overwhelmingly from restaurants with grades pending,our best assessment,is dhu these reductions are the results of hearings.In a subset of cases,DOHMH reports odine that"[talo violations were recorded...or violations cited were dismissed at An administrative hwring."E.g,Ruhaaram Lopmdon Ir firmanon,supra nom 33] (search "Bella Napoli," click on "List Results." then click on dm Bella Napoli associated with "go War qg Stats Manhattan, hooig," and select die entry for "08/o8/xov)(emphasis added). ins.The Bloomberg Administration reports that "83% of grades are unehangedbe Bun o inspection vhd hrarhhg.'IiesAmrant Isna Grating:TIm First Yao,.mpra n 39.a larger proportion of inspemohas hmy be snbjec m scare changes dun reduce(ccs,but do not change he ultimate grade. 668 FUDGING THE NUDGE Figure h9. THE ROLE OF ADMINISTRATIVE HEARINGS HEARING REDUCTIONS HEARING IN ANY CYCLE I IGIUNG IN BOTH CYCLES n Prc-Flcving Prc-Horiug m - 54 Yo Uo f' _S• "' i �jo ao« 1 o ao qo 60 6. qof ro o a60 qa z o Pre-Hearing Cycle I Cycle h Post—Hesring Past-Haring ..11•' th R - 60 qo ao o 6. qo n an a Cycle Cycle The top-left panel plots the poise reducnon in adatinis.tis, hearings of any inspection resulting in a reduction.The panels in to middle column plot reinspection sorra across cycles for any rauurant that successfully appealed its score from at least one of rhe mimpectons,with pre-haring scores on the top and post-hearing scores on the bottom.The right co mens plots reinspection stores across cycles for any restaurant that successfully appealed its stores from two subsequent reimpccu ns.Observations are cellmnod at sixty,for visibility.These results suggest that hearings do not account for the lack of mnsismncy. Hearings improve scores for a small subset of restaurants, but the underlying inspection, (pre-hearing) exhibit even less comlsancy than inspections without appeals or point reducticaus, The top-left panel of Figure sq plots the pre-hearing score assigned by an inspector on the x-axis and the post-hearing score on the y-axis for all inspections that we observe as resulting in a point reduction. The average score reduction (given a successful claim) is roughly to points (standard deviation = 9).A considerable number of hearings result in the dismissal of all violations.In 117 hearings,for example, the score was reduced from above 27 to o. In the post-grading period, as one might expect, score reductions overwhelmingly occur for reinspeaions. The two panels in the middle column plot the correlation of reinspection scores (in the post-grading period) for the 669 THE VALE LAW JOURNAL 122514 2012 subset of restaurants for which the score was reduced at least once. The top panel represents restaurant scores pre-hearing and the bottom panel represents restaurant scores post-hearing. If the correlation were more strongly positive for the pre-hearing (top)data, it would suggest that administrative law judges exacerbate inconsistencies. If the correlation were more strongly positive for the post-hearing (bottom) data, it would suggest that administrative law judges reduce inconsistencies. Although the correlation for this subset of restaurants is weaker, there is no appreciable difference in the consistency pre- hearing or post-hearing.Administrative hearings,in that sense,appear to have no impact on the general noisiness of inspection scores. One challenge to the findings in the middle column is that restaurateurs choose whether to proceed with a hearing.Some may do so only when the score exceeds a certain threshold.The correlation in the middle panel then might be plotting a successful hearing in one cycle against the score in a cycle where the restaurateur chose not to proceed with a hearing. To accounr for this, the two panels in the right column focus on ninety-two restaurants that have successfully secured reductions of scores from reinspection across two inspection cycles. If anything it appears that there is a negative correlation in the pre-hearing scores (in the top-right panel), which becomes indistinguishable from zero for the post-hearing scores (bottom-right panel). This provides some (albeit weak) evidence that hearings eliminate certain outliers.Without observing choices made by restaurateurs and administrative law judges more directly, we cammt definitively say much about the role of hearings except for the following: hearings alone do not amount for New York's inconsistency of scoring. q.Random sampkftom the DOHMH Website The DOHMH microdata may differ from what DOHMH release on its website. In particular, the microdata lack information on the types of inspection. To examine the sensitivity of our results to the classification algorithm and other potential difference between the microdata and the website, we replicated our basic analysis on a random sample of restaurants exclusively using information from the website. To do this,we proceeded in three steps. First, we retrieved the population of all 25,182 New York restaurants via a blank search in each borough. These searches were conducted from February 21,2012,to February 24, 2012. Second, we randomly sampled restaurants from this set.Third, if a restaurant received at least two initial inspections in the post-grading period, we recorded scores from the first two initial inspections.As before,we treat operational pre-permit inspections—but not nonoperational pre-permit inspections—as initial 670 FUDGING THE NUDGE inspections. Out initial sample size was 686.Two of these corresponded to restaurants that were marked as dosed,which we excluded. Five duplicates,resulting from changes in the database during the data collection process, were omitted. Of the remaining 679 restaurants, S16 (76.o%) record at least Two initial inspections. Applying the same analysis, we find that roughly z% of the variation in the second initial inspection score is explained by the first (R' = o.oz). In sum, these results are identical to our analysis based on the microdam D. ComoboratingEvidence fmm Eight Other Jurisdictions To examine how representative San Diego and New York are, we collected dam on restaurant inspections from eight other jurisdictions: North Carolina, South Carolina, Louisville, EI Paso, Austin, Seattle, Chicago, and Florida-"' These data comprise 434,418 murine inspections for 114,141 restaurants, ranging From 2008 to December zolr.Although there are small differences between these jurisdictions, the basic features of the inspection systems are comparable. We therefore apply the same analysis to each of these jurisdictions. Figures zo to za present the results. Figure zo presents data from the three grading jurisdictions:North Carolina,South Carolina,and Louisville.The first row of panels presents the score distribution,with the'A'threshold plotted as a gray vertical line. Each jurisdiction exhibits sharp discontinuities at the threshold. Most compelling is the case of Louisville, which changed its threshold from ninety-three to ninety in zou. The discontinuities track this change in threshold directly. The second and third rows of panels present the consistency of scores across routine inspections,with the third row magnifying the'A'range. (The formal name and method of reinspection differ across these jurisdictions, so we focus only on the inspections closest to routine inspections.) The levels of consistency in South Carolina and Louisville are comparable to that in San Diego, with North Carolina exhibiting even higher consistency (R'is between o.44 to o.47). Grade inflation characterizes each of these jurisdictions: 99%, 97%, and 94% of restaurants receive 'A's in North Carolina,South Carolina,and Louisville,respectively. Figure n presents analogous findings for three jurisdictions that score but do not grade: EI Paso, Austin, and Seattle. These jurisdictions provide a relevant comparison group to assess the potential effect of grading on the score lso. The North Carolina data represpa udrty-six of one hundred wnnda in doe scare. The Sotnh Carolina dap include oidy the last two inspections. 6p THE YALE LAW JOURNAL 122:574 20u distribution. Unlike in grading jurisdictions, there is little evidence of discontinuities at the closure or reinspection thresholds. Consistency,however, is comparable to grading jurisdictions other than New York: prior scores predict roughly o.og to 0.46 of variation in scores. I.asdy, Figure 22 presents results from Chicago and Florida,which neither score nor grade.As there is no formal scoring, we present counts of violations, which exhibit no discontinuities and some degree of consistency. No jurisdiction exhibits as much inconsistency as New York To assess the impact of scoring complexity, we also examined the scoring worksheets of each jurisdiction.All of the jurisdictions have scoring worksheets that are dramatically simpler than New York's, as measured by the number of violations and possible point ranges. Indeed, the only jurisdiction that comes close to New York's level of inconsistency is Florida, and Florida has over one thousand possible violations" (scored on handheld personal digital assistants'"),although the score sheet lists only sixty-eight.3S3 One other conjecture that these additional data allow us to examine is about the role of penalties. New York's penalty scheme, which generates over $30 million for the city each year,3H may provide an incentive for inspectors to generate violations, thereby potentially driving the inconsistency. We examined evidence for the imposition of penalties across these jurisdictions, which we find to be mixed. While Chicago and Florida seem to have stronger penalty systems"' and relatively lower consistency, for example, Seattle officials "seldom use civil penaltles"356 with comparable levels of consistency. In addition, jurisdictions may collect revenues from delinquent restaurants through other means: Austin, San Diego, and Seattle all charge for 3T See Food Vwlanoa RefYvvG,, Fu. DEFT of Bus. & PROF. REE., hnpJ/ www.myfloridalicrn e.mni?dbpr/hr/dommenn/foody fem ce.xls (last visited Feb. 2S, zor2). 152. See Long Range Program Plan:Feral Year 2012-2013 Through 20r6-2op,FLA.DEF'ToF BUS.& PROF.REG.25(Sep1 30,2ou),linp://Floridafiscalpord.smm.Fl.us/PDFD caaspx?lD=6µz. 3S3- .S'R Food Serena tnnPN'oa RePoR, FLA. DEPT OF BUS. & PROF. REG. (OCT. 1, 2009), hrtp://www.myfl ddzliat.e.mm/dbpr/Iv/fornss/documenrs/So22_o 15 pdf. ,a. Sec Jana K Is fl er'ic, Thane Health Grades You See o. New York Rem-moo,Have Been A Weemifall for the City,Bus.INSIDER Mal.26, zom,http://uticies.biuinessuzsi&rxo.v2m2 -03-26/news/3123852gJ_r mu=t-grade=ds-fines. 35S Sae FLA.STAT.S 381...61(2021):CHI.,ILL.,Be.OF HEALTH R.&REGI.S log(z0¢). 3s6. Phnoug Cat Ir,R.m lova,BUpatfmv Skipprd,Fin.for I.fraHavol Inflequem,SEATTLE Por- IN`.[-UGENCJE ,J1dy 8,zooq,htry://wwwseattlepi.mm/local/artide/Resnnrano-inspemoiss _skipped-fines-for-11490.5.Php. 672 FUDGING THE NUDGE reinspection, for example.' The distinction between penalties and fees is not always clear. As penalty structures are a major component of institutional design, however, their relationship with inspection outcomes warrants further exploration in the future. In sum, the evidence From other jurisdiction corroborates our evidence from San Diego and New York Grading is associated with sharp discontinuities and grade inflation. New York remains the only jurisdiction with meaningful variation in grades,but its scores are less informative than the scores in any other jurisdiction. 3v. Sa Dept of Envd. Health, copra nom tgo; Envd. Health Setv., Food Proration Fm, Ausm,Crexea.00v (xorx), http://www.austinwxas.gov/sites/defauic/files/files/Health /Environmenul/fees_2o32.pdf;Food Pmwwn Pwg am Sevice Fm-zax,PUBLIC Ht TH- Ssnras & MNG County (xou), http://www.kingcounry.gov/healthsetvices/health /ehs/-/media/healtWpiiblichealth/doatmetim/chs/2ot2F dP otecrion ees.azhx. 673 THE YALE LAW JOURNAL taa:S74 aO tx Figure an. RESULTS FROM ADDITIONAL JURISDICTIONS THAT SCORE AND GRADE Xonrfl u.Lft A sours r OLINA ..IWl U.E Et.b.:6.81. Eo.b.:13,568 En.h:Ims ImV.:5-.897 Insy.:i,,.96 Imp.:"s, Dates:Vaoo8-3/xolx Dat.:m/a.o9-.yaotx Dat.:s/an.-SAO" B j A It A n n of a-i- lIII 0 ej� r1�Illllll�, .j - ���III o� amtn A o 8. 85 - 95 70 8. 90 mo 7- 8. on loo mrc Or Ra 8 To at: ■ �So 9. loo mSo 9. too ^)O So 9O ioo GycJ.i P9. 98 i.a Afl e9a 96•Om P9- 91 98 W _r reb,L Eadt rolm...pe ri&m A jurisdinion dut scores and grads msmuraum.The top row summaries rhe data for each jurisdiction, listing dee number of esmblishmenn, number of impections, and date range. The second row preen histograms of raw bsspenlon seems. The third and forth rows depict the emsistenry of Tontine inspections from Lou eYde to the neat,with the fourth row fm1smg On 11OPeetio.s m the top grade range.The adj..,,bars mpr colic die R'for euh of the plow.There as discontinuities at each grade thmdmid.Across all three jurisdictions, few rsmnrants receive grades below. A'. 674 FUDGING THE NUDGE Figure Rt. RESULTS FROM ADDITIONAL JURISDICTIONS THAT SCORE BUT DO NOT GRADE In.tASO O)S.IN ill I... Estab.:3.938 G vb.:1a57 Fsmb.:ro,8u Insp.:uam bop.:.,ass hnp.:79.691 Daises:;/10n8-porta Dues:a/mo9-0-m Dues:t/a0o6-Viom oo � VV of B D 9 _ So 70 90 60 70 80 90 so 30 m o Scute R' •I 1] 5 01 z �So 0 90 �So 7- 9- So_....}n 'to F 4d01 UmF' & �' m $bo 80 1. 970 So 90 100 35 a5 15 5 Cycle 1 Each wimmn corresponds in a jurisdiction that scores inspections but dues not issue grades.The top row summadaes the dam for each jurisdiction,listing the number Of esmblishment, number of inspections, and tine range. The second row presents histograms of raw inspection scores.The third and fourth rows depict the consismney of routine inspections from One rycle to the neat, with the fourth row focusing on Oupection,that score above the threshold for elosun,or reinspection.The adjacent bom represent the R'for each of the plot. 675 THE YALE LAW JOURNAL ra2:574 aora Figure aa. RESULTS FROM ADDITIONAL JURISDICTIONS THAT NEITHER SCORE NOR GRADE CHICAGO FLORIDA Estab.:toJ51 fstab.:54,921 Insp.:16,434 Imp.:a38.3, , Dan.: /nom- I/cola Daces:7/2009-6/201t 0 u, n e U a 0 ._..,�d II O01 5 u. �, ......... III) J5 IP ( O 30 a0 10 O VWIanOPS o IR2 0 u o u Z � 0 i J i( to $ 0 30 a0 10 P Cycle I Each column corresponds to a jurisdiction that neither sores me assigns grades.The top row summaries to data for each jurisdiction, listing the number of esublisluments, number of inspections, and dace range. The second row presents histograms Of violations punted at inspections in each jurisdiction. The third row depicts the consistency of routine inspections from one cycle m the =L The ixes represent violation counts.1720 adjacent bars represent the R'for each ofthe ploys. 676 FUDGING THE NUDGE E. SD..cerjor Tables z &3 The following documents the sources relied upon to compile Table t and Table z. For shorthand, we place in parenthericals the cell(s) that the citation supports,with the following legend: m:p gala ory jurisdiction uYoyt uF,:EiucMurn I'1yYTn[LpbNW lC Inspectors n:CnG polo. u,:Publicgrading ze:Posted sign(.id,color) W: �-elL. 4*l'f.4% p+°�i1_',c.:t_ At'?•XX t ng:Point systom xg:MMpecuon:ais. Ii:Follow-up tho,shold v:Reinspection:Days zk:Min inspections For carnple, dw first entry for Albany('Albany County Bd. of RealLb Aes. No. Is, (zorz) (v)') indicates thou die date of enactment in Table z sterns from a health resolution by the Albany County Board ofFlealdn,made eff tivc July a,zorz. 677 THE YALE LAW JOURNAL I22:S74 zDu SITE SOURCE NbanYCuunry NWny Cnry.Ed.o(Heallb Pel.No.ill(NY.toll),M1¢PJ/blog.timnunion.romhablehoppmg Nk�,O�Yayilbanymimpmion-mwlunonpdf(.); gam[:,,up,.now (aa,u.,r,<I:.0); I Telephone Im,mm with Mananm Sion,A—Pub.Heil[h Sanitarian, Gary.H<ilrh k Fmd Ro,,E—L Heahh Sem.,NWny Cnry.Heahh DcWt(Ian.a9,vote)(aj);Tekphine Inr<rview with—i...I Swine(Sipe ab,amt)(ag--ifinw g rhe[rcvuranu nn[e ,a new grade upon r<inspemon); Ah., Gry. Dip, Of Health, A6mu A, Iwpemon Pniu, Neµ mx,.com, h¢p//'raw'.v.albanymu,gwnJdepa[,m<n[.[/ l,ith/e uuranunspeaions /ddaW,.aspAd-4�3(Ivr rimed Apr.9.aou)(ah,ak). Nbw,wu,e Vakri<Sanul6nez,Pa.—Mwl Mix i6e Onk,N .'Fy J.,Nov.ae,p98,a,N(v- ON, is[he a 1ma rcmad w.It— found nvndorvng rhe rtd and grtrn sri[E<.); Auuumnt ftil"n M.IO.Cuv OF ALA,,FM uq b¢pJ/www.i�.gov/enviromnenulheildyfood-safery /m -insp[mon-mala Ilan risircd Oa.i.van) (ae.vg.A);T<lephone Invrview-6 Susan Sprang, F[W Opendans oRir, Mb,-rque ETOL Holrb Dip, Si, M, ao,a) j (a - .Dong rhe, a astanun, — impiox - vain, O,va rzhuparion; mn.g ilw dui Nbnqu[rq—by..rid,a ,t a too-poi.,—nog aya¢m);ALW xiM[yvR N.M.,COO[OF O--$111-it(P)W(eon)(vh,0);id$91-i-v(B)(9)(al);Telephone Wemew with f tae Anluu O.Supwism,Co-unaa Health Pm,Dir.,Albuquerque E.W.Hdth Dept(Apr.v, on)(vl-[IaNying rhat drc aei-padao f<u fr a ae[ond fllow-up inspe[uun,which mmt be r<queand by rhe,mblishmem);Auu�IpR(wB,N.M.,([to[OF OAO]OMi 5$ (mu)(al);Auv4[ay[,N.M.OOoF.F OuOwu[c[5 j.1-6(A)(A). Allegheny TeI Awne Inn[view with Dare Ane,S,Oviwc N,,Jbi,Ott,Pa.,Hul[h ,V,Fou,,Safety COO., Lit,Hae ig.aonj(v indinrirt,,that the phrad ryaam began arouM 1999,ag,al);It u Di:o..Dwt I wo,ry. Or Pow B.*:nlhghvv G,uq H.,-EW[w So—if Rawmm lrvpttion, P—OF ex POST-G.. Ma1 it. eon, M1npJ/www.pon-gavve. .lAT0- /opinwn/pmPttdrq/don,-war�Yoty-foci-irwl-3sab39(v-indim¢'ng tlut the plavrd spttm ha:ban raid Al abom v I—);R<w—Sash in AR hO,Gvury,Au PiOyr COu Heu.iR Der .h[ryJ/webapps.a[hdnq/Rrsnwant(Iv,Wiad Apr.,voa)(n);Au[Oxem COuv!Y,Pe.,H=_uix DVY k h Rn ., m. 111, S (aooa) (x, 0);Telephom In,.Miw with Don-Sdurd,EwOJ Hdui S ,iYim,Alk ht y CO,Heahh Dep'[(Sept a6,-0)(ah,y-i h.ul dn,mmpmmns u<niW.d by[nual v.hdom vW-nfi—in, that th...H-It,for ainspntiov);Pamoa Sabidni.L,t,k Bt..w Itmawnv/vpsoau, PmiBuuex POST-G.Sssnii, Dec 1., mag, 6¢pJ/nww.pv[-gaz.¢erom/amneyse¢iw�hon[ /li1NiN[Sirc-puein-rcnaurm[insp�on:-6a6ns(ak). Ail. G.a.COMF.R.Y Ree..a9o-5-4-.o1(ion)(u.Romig rho t'HeilthmM An"mry be the Georgu Depannxnr a(Humv�Reaomar m rh.'Counry B- of Heahh ening tsi¢agek); Feed Srrvhe,, FutmxCOvnnC .Gw, hr[p://www,flm�mwryga.gov/enrimnmengl-health /OM-fittit s(lea Vbiad AN.9.eau) (,b);T<kphme Ina,view wim Bamry Hvmo, SV,mwr.FWmn Cnry.En W.Hedth(Apr.6,ton)(w-suing Mat tMrc art(acn impsmrs vsignedm food vkt,who wndua food aM mann a aommodauo.inspmiom)(bm f WI k D<wad.i.P nm[6,u it moung 4 i.spa<an));ama Pro OOI BWyq Fl,ar Wi wnrG, GOY m(voa).fry://ww Mwnwunna.ga!ugWrmney Yanra_Pmpo,rd_Bwg.r.pdf 0[ - hil-tig F,aag (Dad O[MiF i-P.nfns pa y.v and l4 mors¢ aanmmodadon inopemo.s .mill,suggem fwd[on.u-[¢ ,61 ifinsp—ns);G..C,wv,K th REW.iY,—(ama) (iQ it, if, 19. th, 31); Mlrm Crnnry N64 Hmh6 Ivpu Pqq DrcnN.Hit ➢mY, h¢p://ga.sn¢g,p,,mn/gmroVs,¢h.dmt[Ow,V.Fulron(wirrisi"Apr.9.aon)(al). 678 FUDGING THE NUDGE SITE SOURCE Armin Hen4h 6 Humm Srrvx. FN Pmetian, AosiMfvnS Gorr, RnP'//'v`^°.aunimexaegov /depammenr/fmd-Pmrersion(ia.,b);GYlor eillinger.FN Sv/M^P'i°^9:I.upxoaiu Mx C'ng Orion,m faun,Et P.vw Tlw.1.11 A.m=a,M1'1//w`^°.elpasotimeemnynews/ci 646pav (Ise einnocl&b.1.rota)(ax-noun6 rho numbs of impwmn in A.,..);Trl,,hone Imemew wiA to pecro on Dnry,Food Prem.Hulth N Homan Sem.,A.udn Dep'r ofHalth(Apr.6,eon) (¢-ru66ennganrz do%afdmeu spent an food crubhrlon.na,mn!'nga5 inspmors);Yrs. b".//www.YAPcoWomun(mar nor oy-1in"Yardh for.boa:toper for rotor vssch rcmw; m -Ddoing (s nu).' rherx trot option tr. a 'Dimnd to dte fil.. mmon) (r-.—W,kr'kod:'aw—g Ford,:and"rhad®ce&day ra,<'in Aron ,V.mggndng Ihat'fow,make u,W.8%ofn ddidow.n);ttT Aorw.C m S 2rtP (=)k)(rw6)(W " ' in"wons);d$219.0=(h)town)(W,ti-'When the mol aunWairw demon'Arr N n uobl,hmmt racds to drmrin_.(o]nanr more wins,cri doll G mndwd.. Tdrpowne Inaav'iew with Sabnna VMaooi,Rene ho,wrmc Aurin Heilrt Dep,Qart rq—) (12,,0;APsnr,Trx.,C ,,,dSton-r(C)(roar)G, '[E]—pt u on iddi,Sear.—Yoa D,pinniped in 11 orMo,D.n,Ana).dr:UW rd r,ehe T..Adminimadre Code Tide r,Far 4 Qopnr 219.SUbrit"tn K IT—Ford EmNithmou Fi...');a 5"r LD(d):Ta..Pend Foot, E,,,nomt Iapcmw Rya. T. DVY of ST. Hu Suvs. (=006), M1nP'//`v""v.dsns win.u.n�eVOMteialiditupa?tinkldenufiv.id6tllmYD.85899A69a (IS, th); Rumour lap.' Sxm.DninAUSiwTws.CAv.Reps//dra.aamn�ugw/daraw/Rnomant inspcorn-Srom/mnv-9Qii(lar vkirod Al.9.tort)(r3). Bildn,ore Wd Comm?lotion,BUT.Cm Huttri OR 1,hol"low"ohrotmne6alA.ory/f"""' Trod (Isst tisiod Apr.9.wn)(n,d): Klein&ChW»I,sopm no¢6,u 13 Gb,r);Telephone Ero w w wth Tanp Taylor.PnW.Srnuonan.Fad Control Seaion,Enrl.,ttpxoon Servs. Bogan.B4,Gry Huhb DrWl(Mo It._°__)(t°-r^^6mung y Nil-,n,,food inxpwors); Dep'c a Aurin. Prof rr Audr PepinCY4 4 B°Id^'°re Hm46 prgmmnu Divuw gr EwirwmmW Hm,F Bunnu ofFrol G ord,9a+T.Cm Orr m wns Orelrs 9(Feb.aooF). bop_//www.mpdf(W-wwit 'I.Brada Got, —rordit%aolow4aproud rCONI1tOL inod hot);TORTEdf(W-nomwv,.low-rnk tonnes arc tegWred ro Re ivp4 Err oro a b✓anDo bazi,;hlepnwo Inorview do rcpnscnu or it do.Bile City HdrM1 Dep LEnvd HeaNt Div. Enordo,00rr conocnlying,here is oma , Drrpotri, ofirpi/www1m.N.Inotth fml pior,,00 u Gonna. ordA Otvn H,-po dr,n hop✓/wrs tort mmehdorrorg /foMclmurcz.html(Issr visi<ed Ap.9.aura)(rl-Pro�'di^6 r^onddY rcpvn IM1u U.iMonnanm ,,,sing only m drrwd soman-). Sosmn Hwlrh Dirum - Fmryend]Anoord Qrumnr.CmOrBosNrv.eov. htp:/MvwuyNRorae60" /«fin✓,d/,gasp(W Zi d Apr.9.tort)lu,W-writing rh n-[Qwd nnb4ahown.an irrpcc¢d at lean onrr per Year): ,,in A D`W",rvpn rote 6f.r H(tb,tr);OMa of Bodge M,,n,,Fro AA,wt Brrdgrr Vn,a^rn^""nPorrrmrrrts/bio%aoEnvimimmenN6a0%=6%roenekigl%ao hit,://—Ltyr gar B<x_ ! R HealtR lnryumn w6oixlobit Gbinet%vorttm,)an89.PW(rr-P,t.nrm'a gx or eig ren on'roper rurwanu,orate,hdrM1 dubs,outage pramrio.rn,and rc—x.wtl ramps):Don pont emot'9ooTor.eov, fborrn gdv/mi4databwtnn(Im tinted Fen.tr. zoo)Im-noting a.r66 rernwaors,19 w—rol Fowl. or Yu rol pw,,and 21 rernadooil ramps,to unman rho food is 93%Miwpertions);Telrphonr arcmew with Bos.Impcaond Sera.D<Y i,H."Div.(Ape f.ton)(It-cvnfinnin'do,inpeaon do not rpunliae in(d inpanoro):Tel<Phow,ro,wroewwitR«p,nenvov<at Bos.lospecdond Serv.Dep'f,Hu16 Div. (Apr,...It)(LC retifyi.6 r.4 ninolow"n r nrrrrRy employed);TaePbo.r I..Mrw war mprem,mn a at Ar Boo Pub. Hnith Comm'n Orn, a6, 2012) (,N; Mg is Fd dnrt - Fxdfrhmrntsmah,eirroe6osrorv.rw.Mtp!/www.r ryoRmmn gw/rsNReild✓mR/nad,.ap (Ir o,i.d APo.9,tort)hi) 679 THE YALE LAW JOURNAL IxxaJq zolx SITE SOURCE Chadune N.C.Gen.Srnr S 11OA-111)(wsQ(ia-giving munry FealoM1 departmenrs aM1e rtaponsibiliry w pmvide poulie health somas);Wako-b du Digial HmBF Depmment MtwtneO.e CaOttn Hcum D-f'- hrtpJ/maVJmbug6giulhdshdepanmem.mm (lass vnilod Ma, a0, so.) Db-adGng up lbs comber o(revauranu sal'nal,fon,..in,, c-term.,ghat_..an.and mobile bond cons maks up SSAl cfmaMnhmena inspected,iD;Rgio od Sundnnon Training and AwAaiwinn Synem,N C.CVY os foo Fa,&NAT.0.rswmess,hery;//appabluelivrdmmgasas (Isse-mond Apr.6,cors)(m-ns ing 66 active EHSe-Alud w M-1d,abueg Cauory);En W. HeJtb N,Dapc of Healsh&...on Servs.,SugCwmn ljosso bn,CATAwen Cuvxn,N.C., haryJ/www.ouwbamwsynegov/EnvimnnacmalhealoNvaRmnnmasP Mass visual All. d, o�vsam, usu,c o33food animpeann inpocawba Mo Unbur,C- wEmdinlodging; abe same oauo sas desn,h(Doti and Iuns an boo,): in Maos,J..O Cman6 yielding u FTE ' noinoaao(a0am, onshe 55afthdo (Apr.S. Innaviewwith rcprcsonsdoae sPoOshModdebung Cory. specsEmwlHdd A M (Ape fi aoaa)(n-mnfinning(waahat) (td)i x do tea apmmh in food iaok iom); (oA 10; Aoum.Cuoe qC Conn (aaso) Ind); of N.C. AunYa.Coot�8A.a66a(varsl(n,d); m.NC.AE V CopE a0A.a66a lag):Div.o(Envd, HdsM1.Food Smnee&wWuhmme Impemon.N.C.DVT er ExvY&NnT.Rsroup,haryJ/ charmeh.mg/mxllenburglmunryMdcbCepartmanyEnvironmennlHealdyons, wSemas /Ossans 'Osa/Dmmnnn/DENRgooJ F aao) MisF Bakes, A R Press, Sona Famurenv Gn Q�ureGe 'A' GMe. SULSeupx Ptm' (N.C). Feb. q. ao F.ptaor ca.mourn within as Rdsmr,Funninisad emanon is sed.....J. impeaosc,.,morn wicM1in.o day.w nuke rose rtio mud viohdon.s ued...'). Chiago Jny Pmemm Duuun-Fd hupsden rypapn8 yppn, C15r pu.. hap,//wOullOaa,visited Apr. ad); gein& De Wail,supra nae 6q at a5 Ob);Telephmn Jonrview w F,Pamek O'Connor. .upaaviaoc,FOM Pmt Div.,au. ,p',of Pub. Ha4h(Apr, rs coca) (n on,,(add inspscon,a5 ofwbom ilw perNma Pool inspmions,and a rvpervvors,and admadng dsaa 9o% ofinspenos'time is spanaonfaM;the FTE u uaton furtMraasumes thu mpervimrsspend 596 afabe'u time mnducdng Find mspoctiaos);Can.,lu.,MSm.COO.,"o(b)(son)(aQ;Pad Punnisn Dwnm-fmi ha naaom Repro^g 5,..C r of Cxt btaJMbapwr•arvaubi=ga.nq ,gnahhu,,xxsi.0 mspv,R.O,p(Lama vuhM FA.p,son.)(,j);Fml hnryenonr,Cm of Ort DArs POPFu. haps//daudryofchictgo mg/H a th-Human-Swioey oW-hupmims/44n'sT F (last vi inti Sept as.vou)(,j-providing impnu..dm). EI Pam Rp-t of Pub. Heahlc, Fad Impmon Pmgmm, Cm' m a PASO, hm,V/ nvdpuotam gov/FalaNfmdasp (Inc union!Apr. 9 ors) (u); Cao. 9il4ngeq fid Snfdy a Priaiy; (vpecnnv Cao—1 Ondne in/ammp,Er P.T.,f WY 0,aoaa(ab,m-rating eighteen impeewn doing Road impecdmn,ad);Telephone bucry cw w@h pond Subboky,OF, ,EJ Paso Cep'i of Pub.Hnbh,No.Ann—Can,(MaF.to,aoaa)(w-verifying rM1a,impuma spriilue in fwdimpeviom);Duclaimu,C, OFEtP.DV Y.1 POR,H..M1[ryJ/www.elpaaotexss gopM1ealatc/embWbmen<impnion diulilma.asV (lait visited Pb. a, zoo) lag); Ieusu Shubinal i,Fmf hupvuiau MI B'kiM;0 Cautery F Iruprtms Manipn coon£Manu,Ee PAw T., Jul, r, acoa, as I (Ad; Aaron Bneanwnvs, Hmv Soft 1s Your Fd?SmeA Fd Eibblad-oe Ivpannam, Ea PASO T., Nov. 4. an, M1apJ/www.elpaemma.mm Ai 19,15L66 (di); FoodroNmem f EukInspmum u, Cn a pu. Cuu r OF POP. H.. M1apl/u'nu'.elpnacuas.gov/ho1aF/eaablisM1mena impenion.up(Wavrsiud qpc.9,aon)(al), 68o FUDGING THE NUDGE SITE SOURCE Georgi Inuryrewien MnuW ft,the Rub and R,,IO,nv Em(Sn Oat(A M 29o-5^4.Gn. OFF I OF Pue. Hfu i (son), nupJ/Taahhaue.ga.us/pdfs/m ul/Food/RukVFOEIFS InrespmaanoMlanuil.Pdf(sa):Feu.1E—.Ga.Dept 4 Hunun 5 m.,Effaaiv<De<rmber s: N<w Food Sella Ikgulnians(Nov.pE,aoop)(on THE wi,h auahor)(aa);Ga.DVY OF Cure. Hcum.DmSMx oa PEEF.Hw.FaoD Seen,[E awLii lIxse[enUu Ra"OE"(=oa9) (=b,EE,=d);Gn.Cou>.R ,R[es,a9u-f-a-�O(_I,)(u,IT,ae,If.4.ah,ai,zit);Telephom InrEr w wida re,waaauuvn az Ga.N l of Pub. Heald,Emd.Hdrh OFF,(Sipe a6,m'=) I, nodrigEbn fee fw ar<ivpauion var:eE by aoun,all udnudng a range ofg5O-aoO). Hemp. Dep'E of Health EE Hunw km.. (fmnnl Infmmai+n. Cm oe Hous.. hnpJ/haunm.a .geeov.mMmediyides.dm(lou vuiO,Apr-9.aota)Ua,ib);T kphmu Inruview wiEh Bom Hsu,Supwuoq Bmeauof('umumer HE,,h Sem.,Houa.De,E Iall lb C IT---Serve(Apr. ,a. oaa)1.-explvnin8 Eha"metwiduils modus impwom,Of whom I are mpersi wa--o ,Oadua pmopening impe<rims a foW esublishnuno all aM1er M<dmsw,a Psi QUAY Oily had operadon maugns,and 9 T.rondo.Paul inspEaions Lang the E—Er;PFE Em1111me5 a.mu dose 8 individuals On aveaea spend half w Ew AO"ia If 61""1 on food imparaon.l; Hous..Tex..Coos w'O.5 ao-vola).R)bmQ 114 d);Dep a a HnlEIE&Human sawn,Cmoa IT..,hErPJ/h-OEEE Pas vhiOd Feb.f. Ou) (11). DOpiI wiapdng she Texas Faad Psubliah—, RUM. Hoot., Tax.. Cone OF OFOrt+N+a'S 5—LE,IT...dors na E Poim IPEEm"I-inspadng I'mss —I Thr Cepvnnaa a Heabb and Hmnan S rv' Oo-as—an WO-1""Ma"in whkh Ir am �awawa OnvalE of to f.TMse ndag,v<noc madE FINE,Tekphao<I.Ewvicw with rcpmunuos'E rz Hous.Rp'E o(Heild�&Human Sem.(Ape 9,moa). larimas CuuE, Sony Bishe,fvpsmu Edin(nw Sa-f W. RawOO,F TCO.,CO., x. m99,x,,(aa-ProvWing rhe.Neu mord we luve harm Of6iE Baum);Tekphna Inrxavkw With Jim Nwa..I-P aor SupemEo,Isomer C'Eq Dep'E of Heilrh h REEVE,HOW. Kahn Sem. Div.Qay.19.—)(aa-suggemng duE Ehu Oat"b<gan in E999 a r000);DEP"a I,h H EVYE.'=d fvyervn F^^.I^xurex CaOxir,httpJ/lvimermp/hd/riolluOnasP(Wr —imd Apr. 9. sou) (ab. u, ad); DEpl of Haalth h SIEVE. Romys, Inxnsin Couxay. napJ/IadmEcaryrooNa.p/amring.hran III anvil A,9,arra)(EE);TElEPhana W E"iaw ash Jim C<vara(Sipe 26,mu)(u-roEO—"En.RIM—P impeaions art w amur<wm,b.— andanno,Ehangasong):TelEphom ImMcw with I(arie Sill,Impam,l r,ar Uq.DWI Of Haahh h GIVE,EnvEL.H.16 SE—Div Um.E9.=au)fag.sh.ai.=i);D,, FxalA k REEVE, pxd ytn p,_.luu¢x C trvr,hupl/—hrima.oyM1aldJ<hy(Oodl.p(WE vulrm APr.9..ou)(ak). S 6M THE VALE LAW JOURNAL e¢a;5J4 IG I3 L m M I.Whn,I<ner rn the Edi—.lx).Funs-HALL-11a(zone)(aa);Pwd E2dluhmana ptnon R<pm,S. Nara. Hcussa Din. Lenin), bnp://wwwawlh<mamdahadtheiandm a, —jar'eioladona are gsoupN mgaher am'emiainl vioUaiona,'mr;&Neu.FwoFewEsiuuSlwenTp m.e-pav(B)(vola) i); Teleph.. Invaview wilh rcprcltrande aI Ford Operau n FnM & Barrae ubluhmmrs,S.Neu.H-hot Din.()an, on)C.);Fn W,Hra,Div,Permu b ,or Snmam S. N.. H. z DIST. (mama), halm://www,mmhemnevada hedlhdinnax.nag/downbxQ/eNeh-he-rrhedule.pif Jai- leading th< Fs(m a mlupertioo that results fiom a downgrade r 'C'); Reuumor Ivpectiwu, 5. N[u, Hestix Dlsr., hap://u^vw-and.alg/rc...arm fird,r h,(tau nsissd)uney xnaa)(rt). Wa An,ka Enrol. Health.RL W @ Irepxron Goide,LA CMImm DVT of PIR.H.1,xa,p(May on),h.pVlp.Wih<aL6lacounaY go%eNdou/ReuilFaodlznpe<uonGuid<.pdf(u,mg,Ih,aq ad far the prpoces of rhes r<plr,"Section I-and•Sectim 11•violations arc mnsideted-conical noUriont aM 'Semon 111' mladona ue mviderdvbUtion'); hW Fao4ry (Rumvam/Mazke)Rmay LA Coumm cont ora TA,H[uixnh,;tLlapublirheaNm.nry/rating (last wOr d Apr.9.IoaIJ(ea-namlg Jur'Whe court of la,Beach,Paaad<n and Vemea iwra mbe,—rcuiMnod m'lidrs,mi);LA CGomm.Cu.,Cep[S...ovolo(mn)(I,);lin& G lie,mamma note H,am Ys(Ab);Ann Qow,PrurraL doe Harder a,J,am, mr,E,,,pay,Gar Lounsu;&IF D"at",at i8,18(It-ondng about ap ReW inapevonl;zeal-talo Anvul Repm: Guti1{p a Learmw L1 Oot,o , LA Couertv DEP1 oe Pur, Hm,,JrLI 19 Artp://www.lapub6<LdrhulFJdmpalualrcpanFTo9-ao-yAB-LpdF(Ie-noting rloar no—M, I,- of about Ivs.000 tend inapution wea< of.—.); Telephone Innrvssw wirM1 uprtvnnliv<at LA Crory.Dep'I aPub.Health,EnW.Hdmh,Fwd Emblishmenv&Evens Dist (Mar, 19.mon)(Ie-mntionen g Jut ilupevov do mt dual—dou food me'rarone); Telephom IntemewwiJl Otry,Envd.Health Speaahma aM Supemaor,LA Gry.DePtdPub. Healm Om.n,nu)(mc,d,ah a;,ale-mung that rcinryeeuon depMonwbirh vio4Joos were orN,nor die number of ping ai,A);LA(Mpaar,Gr..,Cee[ora Owwantu 55 To, D9,45(mon)(Ie,tt,ag);Telrpholn fierem w with 4S<nior Inspewq LA Gry.Dep t of Pub. Health(Sege a6,lou)(xg,all.v-31.13161har a...owner dwadaLed wN1 W,%,,iniJal grade may megoear a reinspection within ahnr cars ofthe rn atilu Inpemon;the owarr,m,ar,wy the applopnue fee -dmin m1 &,a of d1u .,—, and the Iunuran WI then «ave einpevion within len days of the pan yemLA Ceurcs ); y, C.,, �N, or Oeowje,2 5Bq I ILU(Io;L.A COmnr.CSI_,Ommra E9I-oop(1998)(aa,xb);Fum,r,A.Rmmlrmt Hpienr Gmda, Txa Tzwceuavev Pour Poo1H%e. ha@://wvwvuupr<nrypliq .nMararlya.eniPg(a);L.A. 8.04.ne(-II)ryJ). 68z FUDGING THE NUDGE SITE SOURCE haus ille Ruwu.rvn uM Fd Serviu Mon. Sean. Inulsvugv.mv, harp://www.buisville4y.gav /HdrlalRmaurvdnspeaion5maeeham (I., virircd St, a5. xouJ (u - noon, rhu '�ilnspmion.aremnduad by L°^C000 Mmo Depumwm of Pub4c Ha,jYWdlnev lel HYSene Pragtun7; ImircBy)e�v(I ,moo Merge.luu0t2)in .ew,hup//wwwLxlonon of .goo/ovilll .ni J.ff egeebmr w); Wide Senn.x5.-I o -doaibing cansolidariun of Wwsville aaW IeB<sno^ Cowry); Opo Oma Powl m Auuurav (vpuAr Dnu, On, To, )(.b-, nnp!/Wual Iouisvinofliniw/darveyrnuun^r-inspewondau (last naiad O tl fin vara)ntm Bad on aM number d,entont,Uojddrcse ,p t aftb.lin ane dun,di:E- (M from G_)(on Hoyd,E^W.Heilah Supemsor,luuisvilk Dep'a o(Pub.Heald.h Wdlnoa (Mu.r3.co=n)(o^6k with auewr)(m-tB-Ym^g D(^Mime ivpecwrt avignd m(rcd hygrtnen TelepM1 go (A,, i wino Gnahen,.noFitn"I ift. Heahn Supernsor,Louisville Dep'[of Pub.Healthxtionbw (Apr.D,,it (.e-iMiamgnd dwe arc amaoy H.noloo asngnd m FCW FYo¢sdonburan, o,eirumeM .frcd and lead i.Gito,iom);Gil,rvpu^orz4o(ad, Nmaunna Eublunmeu You. 1.'.—2)1...ii. bald//vA,A); isvJleky.gov /Heald/ab On n,,Ky,,Fit. on(Ian Wind Apn.v,cora)(u,d,M d.ag,;C»)atE fornmuk A,FnniaNCm..fo t, Bo.ae ww<an ion It ae woe'T Ns Fog Eiji pq;Obkn len xaalan A,Enmity sea+.,read P,aNuNuma fmlena.Ree*+.Det NH etre.xMam.mw.atAVMnH ar'IM.(_.).M1up:Y/cots.ley.gov/IJR/rdonlyrcs/946Ba338-CaB,F-46E5-B9CFRFC6FE6))oF?/ o/NEWDFSaoeRETMLI'OODINSPECFIONFORMRerimdomgaa�apdf(rg, it,ab,,ad):gen Neulauser,M✓4ing the fnc .Rnmmnna Adjwa to Nem Hw4A RoPeeoen Ra itil-CAUPv>-)., J" a5, -1L. as A' I.- noun,mono tiate absurt for resta^rann am.ing beWr 60); Risk Howka. Hmld D,nnmmt Rnnnng Rmaurnnt R,oin , WFPL N.. A.I. aS. t-11. M1np//archives.wfpl.org/mu/oe/=L/nuLh-d<parzmenoaritlng-ranunm-rano, (a( - natio, smmPa^Y�ng imags of sa ,Bade sand oiditint field(ua asort);Telephom loandew wish an inspaaoq Food fir,",Pmgaam,b^irvJle Meno.Dep'a of Pula.Health jan,ao,cora)(q). Mumi/Dade Div.of Halls k R<stz.PuWis FW Sm^a and FdSIR f^=Peavwm,Fu Dfp,i ,F Bet.&pent. Cou^ry(FT) Res.,hn wwww.mYE°ndilictae.sem/dbVt e/ka(avp 4M1nFN.DFPtw—.&P-' Div.o(Hmels k Wan..Raummur/F".d Me_ Mord-T ,F-serviee.hmJ (leu „cry://www.,M 12ida4ranm.®On-b --iron En d F<AtF.coca)(.b-basilac—in o, nwnbenbeaonnoConoot j-.n,.uW a,t,aorxirs inning Miouane nuc o(wiqus addrasn in Dade-R.' and Hmk Cmmy.,Mlpsilsvka,01 q); t aFi fmnn ntf. M.Leaana,On,Manager,Div.of Houk I Rem.,Miami o s®ory Fla. to mpfB�x&RoL Regnlan^^Qa^.ao od —.it); Di,fti Mine-mF,not. a6saoiwims m impm « n.nnale, ane gala): Dw. M xoma k wn.,Arvmd Rrpprt Fu.Det oe Bus.it enoe.Mn.tt-n=(aa W,hnpJ/wwr.mrfkddaliunse.sonVaw Ihr/npoeWannualrcpaeWdaarmuna/uvo.o u.pM(u-eeamainiog tlaa'PWlis toad aervirc impetnmi wart eb%ofall inspenians):Fu.Aowrv.Coo[Axrv.0.6sCa.aoa(e)(d)(t)(mm) (d):Telephone Inamiew with W Cu-, Div.of Houk h Rma,Fla.Dep'a of Bus.&Pm(. WBolauon(Jan.ad,ao.a)(t(-venfpn6 dun insPmTn^rtaulndo non have to be pond ins lk At, 683 THE YALE LAW JOURNAL 122:574 2013 SITE SOURCE Milvau4ee IT.W Cep e, Fxd E-11ohnmt Ivpemm AM^n-'S Sm , Cm nn Mrtwn0re3, hVVfiimdapps.amil—tot nhn/searcF Rp(Ian iiurcd Apt 9.m,a)(v);Klein h CeWvl,mpu tint 65.at m(6,ss--gong n inspenars);T kphone Interview with Enni HdA S,-ulm II.Disuse Comrol K Cant.H.Ikh Dir.,Minot Ninth Dep'n(A,.ax,aoa)(x--ming m -spars, a in fand and a in weigbw and mastic,with a oath,find impennn Am i'ryann6 nation and pertin6 padors, and uplair.in6 aha f 1 i a,e mm also impxn ma—ill—and ninx prcvanuon 115.-22; FTE esrinno-z manic'Imn inspectors oon—ti, al nni 1--Ba%o(&n, n find.);F"anal,'Ntrd QuumrµpuwnVaY q MH. DVT. M1nry:/Finadapps.simiLwi.uyceM1ai/G91ReaYPojnspen:ana (Its anitd apa,, 9, aoaa) (,d);Telephone InuM-wish]at TnnersN,Emn,Heahh Spaialin,Dan—CDnuol h Emu, Hal&Div.,Cil of MilanW Heil&Dep'I Can,a6,aoaa)(J-verifying 11,inspxmon mpom do not have ot h pond);Smash,Cmoe Muwnarcu H JiDUT,hgpJ/amdapps.uval.wi.m /ahR/ssaaA_by.jsp?roMh.889a}o600a[yGM9VPd4(sspzl¢9RBOVa3MVVA,(Ian visimA A, 9.a-a)DU Miciuip% MwwipiSwsDepmmeuofHmbh AnwwmNnu Aumoraulrupsmnn Marg S)smn„mpro nae 41 (v); UMrnmMinT fmf F Joy frupdim Cmb, Mm', ST, DVY oa Hutix, htp//�ndh.nrtgo/msMaiW srssi/3a,poa.p,J3l.hmJ (lass riaind Apr. 9. aoa) in. ah); Telem m 1-wmew n5&mama--a n Mia.Bow Dep'nnfm);T (Sept I main pg-neon B ahuranonnas can iod Prowfinma•C'rn,DcVtEma ofHeinspati.m,anut), nemwvi). wi&lonn Luke.Dir..Div offwd Prow.,Mia,Saw Rp'mfHeal&Uan.a9.ma)(xg,al.aj,ak). Ncw Yak prift// /uprcrwn f krs N.Y.C. W4 M Aff. h Murtu HYclma. M1rtp://ww 2. (1Bot/hmJ/dob/hm�yai✓�Mez nandlaanormalaman .wn)(a,ab,nl);Ca.. mpmtatnob(se-mdn6a m-enaily1.%oandptnewimpairs);vo .);tnp-nml Cn an Y ming dun appmsih,R Y ae%o(impamons were matmo);FW 5 aM Limamm. SamIDMn:PW4s Hm4.V Wk and pefmnbiloy,N.T.4 DVY oe iI A k Mumu (o, - an, [bl'•+ . VnMih Hinta,Lnn�mpea(�up.shmd fain -coed Apr. s, xora) mo - and, n (ann” Hdrh Smiarimr] art em mind, ttry art .ag.AaRy a;a-1 ao iia en.polmma•);Or e�m aea.abol((tin a—m mb x,ag.ii,x, is ah,d, ak);N.aswoh Ran id v l ogro9 fa u-adding uP a, 8xanean vumber ofpoNn pxsibk far eaeM1 la/Ronn);Famvatnno-,ton Avulu,N ( Ovsv WTA ha�J/nYw,i-J_osoma.mm Maldyandm nnanspsman); ,,ca aw-Jon flan i;aiaed sept y,mx) n-pro^ding a aid ro,.)I, d ina tan.,n. ; .1 Rei.�bab(June g.moo)(v)i U.x a6o8-o9(awe a5, aoaa)(ac,ad);id v a6aq(ac,rid);d u abo)(x). 684 FUDGING THE NUDGE SITE SOURCE North Carolina Na N.C. Sess. laas F3S-ID (a); zy N.C. AONIN. Core IBA anon He., Nott (»-nmv,"vision ur ,Be);zJohn CrcM1nn,Per. u M.ry Cleve S.waztion Gr 'Spar", Gx[nsmow News&0. .IL IL,III a[B.(v-quoting m dhcil"aril dr-lading sy+amr ,.hmia use For So Years);UWv of EnM.Heahh.Pod Sm¢£vupEs5mma(rxx ,N.0 Dvt OF Exv T&Nor.R[sow.¢e(ano8), hap://www.deh ev ncus(mageJ( mpeudoam /DEN0.%aaq^oa%aa(�yr^8)(q%£Flip")drc(af-re,ad); ,N.0 M...Cow. xA.aaaa(a) (ami)(u):z5A N.C.Aowtm.CODE 18A,6-(moa)M;a r"r(h)Cry ah,d-wdng[hu a rci,rarn'For[he Pr ,eWtt dniainl[he WPhabrfid grad`'museberm o cord mdwauldoau[ within �S drys of Ower);M1 d,Baker.S-Rammwer G wQP/o^rrxPIl Goeaaiwpea. Por,"G.). Feb.ap,-r,h[WJ/wwvs urypostm -'m fi -APym(ai-'Theuuprear, ramp wirhin m den m ndl sure me nidal vio� it fixed....-);Tdeph-Inamew wish J.LYn^WM1an,DO'd Sur"imq Fd h Fa[ilit"s Eaniarion Pmgrw Nan.a,noes)(y);aoAN.C.Aouix.Care O'OH(a)(a)(mm)I* Philadelphia EnvJ. HodA Em+..Fm!pmanm.Cm of PHLU.,hapJ/wwx.Pluh.gov/6aIdJ`^vnoamen[ /(oodPaorecdon.hunl(We Onard Apr,9.rorz)(a);Press Release,Pa.Dept o(Agdc..Agaadmrc Saapry:New law Improve.Food Safety Cranugb[,Addr Addidaml Trarapaarany.Uv xaaxV m Insp[mone (Nov. 3a, man), hap://vww.agialmrcaaa.pa.ur/pmvl/ttrverp[?open .aPbvSllU-�o9a^991 mdr�a (a - oodag rho[ abe Peanrylvania Degr[menr of HraW aodN,'hd a sarcwide sarAud fi.,,,peaiom in waaJ;pain&Dewail.ny"w[e a1.a[a3Ida, rc);Pa�rick Kehsm&Jahn EUNvm,Ciry in Ned oflWaurvulvpuws.PxvA.R+QgnuA Nor. (a -fbe ciq muy.mnJa^ao-n-.a/aewyal.Oma_[_raaainapesdaaar<,.irial-Chimf n Frodcns(MR, wiry M1as Saimprcmas.'):ie1Phaae I(Ir-lwndh Bernard Fntrl,Chie(of Fwd to an Phil.Depl-ar of Pub.Hdrb(Mu.a%zoav)(ac-vedfpng`la`inapaaaaa do nm spat C, iald, (Ane);Tdephow Irveniew wird BrmW I os tltie(of Fw3 Pm., pnJa.Balm Pub i.p (Apr.aa,aaz)(rc-eTorrulg`ha`i Hera- spend¢Quad 8.%of [heir time on(ad ivputionsJ; Aan Sa,ro r. hcE up Ju Hera m FkNdplir-fR-o7/ ) Irupxdons, enun. Lu2uuT AuB. e. zoo9. h[WJ/>[dcla.id- Fenn tire-oba-/mwa /m906oTa_ Tele hOlay-rc auaanoivpmoaa-food-""Rema"." (d- nodes f od S"M" So-., Tekpbone ln[eA,i..wire Shen Mama,Program Manager,Bweau "olid—); lab. Eem.,Pa. ma d Agit (P^r �8, ao[a) pd-moo., ing rhe p ofPrrb. mi dr(Ma: Tekphom Inaerview wi�r Bernard F.-Chir,o(Pwd Pm[.,cdon W-)dPub.HalIh(May oaz)(r(-red, ah,e, GO,OF RO.-dater,paidDRaTr OF INN. rca . h TVd PWIFeIr rvo. ftsy yfery Inrpertian Rrysrta. Crrc os Pred H Des'T oe FM.Hex'". hrry!/www.phils.gor /AealaM1/mvimnmenyF^WEa(rryR<pnns.hmJ(Iv[viriad AFT,9.mu)(q). 685 THE YALE LAW JOURNAL 122:574 xolx SITE SOURCE No.— (m1 'n now` Gtfeumr Frvvn_aaaaa—S.. DUY, hryi/www.nurimpagov/ENVWV nufiealayPe Seoaing,upa (Ew wind Apr. 9. anaa) (aa, rc-dowbingvol—lanow nd-p ri,lonkcrhe—IonMrltie Nd.le,•y,_,, , iry vialaaonf a re aAl"I1-0.da,lido mand•paean�ndaunn,naonr u.aonri'I soon idauon:.As o(Apal omni,rbc g—h alms.in Man H Coma waw oa Sop :an, a,npa Hartµw o Telephonel.Seem whh Bryan IT -wonfinow,,uS,, ,ar,Qon Regwn,OJ,of Gry.End,own,Cep'r(ApewTmm)(a-mMnwgJ5an.E 4aw,Manc,art Bo-0$%o( Se-owdme an.1elew on )(;TLk[,A,In¢rnew aw,Jon,Inspeam, nef,ha Cmy.FnnL Serviees DM'a()an,e, awa)(d,ef,an,A,el,rk onunion enat a rasuwant will nave a fdbw-up do,ventwgon Jianaw.. mmarc, indryr oliouyIa a aa-Ewries con ;glen the —r gea of au a Mwee,eeg,a h t On,. uMw am.olIvaD a.dn -m ciao donne a raomanCf g nno, prcu R Incl M — Guy. Enw1 SN—Rpt,Ao •A' hr FW SO,, Mvimpa County EnH�wunenW Sewnrce grmoumss Iu New Fuad Inapuaan GMing SYwum (Oa. Lary-):Ren Se— ba,aKewe}Fanl%aoGeadingA xnuwn,,e Oe,aa):End.Rrn Rp'r.Fm(FmtlSdimntr Smmn.Mwrom Cwrtm.MpJ/wwwmvimµgw /EnvSonce(lan vip/business_seaa1,). ss_pan._dde.FootlxwFmb4annmua%ao5ued�&as _aH'e•Feod(Lra veiWAp.9.xma)(q} San Bemudiw Se Ghoa,eupm norc eJ,at B3 (fa);EnW.HOIA Serve,ABC Aeby Awl loWne , County COUr of Sud Brau.wpwo(Mar,-,I),ntgJ/wwwabmumy.govla.nwRpw£nvimnmmW Heahh/EHSKaoDornmems/ab<_aniLjood_inapemonfuide.py(ab,ss,ad,ae,a(,nig, ak);TelePMk law—wish Staph i Div,of InW.Huddo Sam.EaVt of NMc Healrh, Crary,MSan Bemadino(Septi 1.don)(nig-mnfinaing ewueanu—improve thdr gonia .,on ec nspevion);Scor VaMarne,Baud Alone Buq A-BO;Rnnryutim Time Trimrndfew 3o b In One.Tna Sm(sn Bamdino.Cal.),Jam f5.aow(a))� San Di Fd FaaV bupcnon SmrtF. Oiur OF S.W D¢co, ho V/—sdmunry.¢gov /aeNLldryfiYnuo.nmd(Joe, ebd rape ii.aou)(w.b.If.nip;E-ani CwreaPnnaena Wm Ban Huynh,Supemising EnW.Hdtn Spedaluq Forth How.IN,San Diego Crary,Rpt o(EnW. Hglrh(A,It,One)(on file wirh aThor)(I,-wrifyire,,69 ETISf,5,mMumng fnspcmona,e9 mndanw,iupocaov(tall cine,and aamaann 1l in d awe ryent an food iw,cdwu); WJlumf h gemnoini,sup.mn Re(d,aa,ek);SnR 1.COu .Cu.,Ceoe or Bee. Dwuuxe¢v 5'1 nO(amd(11,11,a.,nig,ah,A);Teleph—Form w Weh Ban HuMh(Ape on, ao.a)To.an-wMmuog dot rcawnnu Aat radon a'B'ce.-C grade w'JI be eeirvii wimin m don): Snn Di o fid kas'ry r iynuma So A 00, OF RO D., hupJ/w Ola—my-1o/H.w (last Hated Apr.9.aou) (al):Runtl Basi Ani OFnawh CIA supu ewe.H,at 6,a9(.B,ih,ab,aS ad);)ort W-Head h Vojkan See,wi Sem on Pws SI'a— fewwwml Sony Tw,Aep.n. ACT G Heu,'m Dmec-ICenrt S Oma 14-.5,aou).hot,, www,heahn,aa.gm.aw4nnlebVa-rendfile6'fnphfid__ada9J{8�69+sm.(2Q; IXP a of faii Hedeh.Fixd FmJip F,t SdWWe.foam'pe On D.(Rev.Aug...aou). huPl/'www.dwmry,m.go/dWfoDJ/PdUpubli®,f hdde.pd((y - now, an Lowly ¢fee rc-BMv M$.*Telephaw,Iw iew wish Co.Pomboa,A.w anow Die,San Diego Cory. Dap',of Er W. Hwleh (Api ay aou) (aj-alwfinng rho ehe he ap,U.ro a wood reinfection abat rcalu fwm nowompliwn on,any uinsp:aon dau u a oa-grnlog impeuion); Tdcphom lam.iew wide Celia grog,Enval.Health Spedain h Specialist on Dury,San Dago Crary,Dey t ofEnW.Heal6Oaa p,aou)(rj-fms arc oNy Paidhrrne s.wndrdmpnuion). 686 FUDGING THE NUDGE SITE SOURCE San Francixm S.EH ,,,COoe 51PP)I_)pa-nosing ahs fart al-,alliblish—al viral abe Cil and Cowry of San F`­ moor have a pelon,ham flu.Copan—of Public Heal[b);ToldS*q Pmmm Ina annow, SF. DVY os PM. H—. hap://wwv.sfdphory/dpNEH?nod Qnspemam.asp(las,-'Od Apr.9,loll)(ab,pd,fg1;Tdephone Interview with Gv O'Wky' Manage[Food WIT Pon.Pmgam,S.F.Dep'f of Pub.HeaWn(AP'.6,anal)(¢-n nfirnting aa inxpumrs,and ahs ioapacwrs also inxprn abam 6,Mo(inal sal lishmanil,nos hundrvDOUR', 96o mbaao strops-nun.of whish are pan o((oM esub4xbmenv-aM pn shopa vial nvemighf lno.h Asswning MI(af mM¢o shaPx au alm kW esralotahmenq float anaMduiixnn nuke uP F.%afestabinhmn¢mryeaed,Ngguan,v PEP(,iingemrs):SF.H ,rFl Cam[50,6(C) (d-dixfvssiog the Symbol o(EaeAmce):Fwd Fafy Prgmm;Rermuana Safny Smaa,S.F.DVT oe Pue.Hau.m,h S olo wn'afdph o cl);S.F. COO-SS 411 F(A)-(C) (Un(if. iud APc,q,aou) (,onhni efi .I naa anotEvmnsse7: al -it Coale 551s6a(A1-port);.naringfhsahe SymMl of 0alknre moat h pvrcd a well as fhc irupeadaa repro[): Mls'mNIPw. hrtP://'nissi.blaa g-vri'lanaisc 0,evu nuh aS.-2)Ili); (Isf Hliad APr.9.aou)(fa, Maaa ea, Soo Inurview wirb Lw ,, OF Y . at.vosa)Ili);EnW.H Bnvaha Bwhm N.xme as Soon Mdres. S.f. DVY or Pro. Hewt'rx. hnpJ/dph-¢sawz.sfdPM1-g:M` /pk/aop h.WP,p'13a:1:.F6alNN6696(Is[dated Apr.9.aaal)(g). Seyde Ph, pontoon P,,om, Pan, Hrx'mSuisV & Kate (bPxav, hap//www.kinpontal gov /lwlfhserviacyhrolablahWlllofry.-PaOa-ridttd APr.9,aou)(a.);E-osiiC alnolk- withPhilWinona,Holan&Enwi hi+ ,mror lll,En W.Hal4h Som Di¢,Dept ofnb.Hrolah, Scalia&King Cnry.(APc u-al.aou)(on(da with author)po-soling 6i wall hapumrs"'1 32 PTE,hod m olors):Fink Bald lawn-Agmm.Pro.Heuix-SvirV&Kwc GOOF"(loll) M1^PJ/www.lun&aoml'.6av/hedahw_oWhldMeh%TaodvfFy/Fba111 ..'Nfaxd.BBomW-1onO ii /Iiddi/WbliehealfNdocwun[r/(cadsafery/vonNskBasedlvpmonPmg'urtashx (N); Tdephone In¢anew wiF,R.,BYmq Hdah&Envrl.lnvewgamr 111,Enrol.Heald Sem.Div.,Dept o(pub.HdN,Swank-King Only (pi on.an")(af-waifpnB flan'rhert u no mandvcd WbGc pooling u(halS,l n.oino.); Canmm Do-Afros!mf Sgfery,POP.H.-Swrni& Kate GOOxrc, Mp://www.tingmumY 6ovMakboemrox/bealfNeE;/(aadsahry�odfW.voPz (Isr A."'A' R. Oil)(ab.fg.u):FdFxaALHum Bupsua Fep*L POB Hesuas-Swine& lap, Caoxry (-7). hap://I,inBnounry.gav/6eahhxalvivaa/LmhNeha/(oMsakry%mxRcaionr /%/4media/hnkNPattiahnl�Ndoamsss/faadsY ry/n.pamo�Ja�m.sslu(Fh-adding OF drt aaN maximmn number of Poinas);Ewd 'omew,ELr4 Pro.Hw.ix-SYnnaeIF Kme Wurvrc, haP://www.dautr.rog/Govrmmrne/Food-Evubluhment-hupeabn-Wf{rc 9n' m5(las vilmd Feb.q,won)(qY South aolhoa New Aomeam�hupwian Sybn in£Qn in SC.,Fwuxce Mov.N�xO News,Aug a9,w6P,a[5 (aa); Mlmk Bmh Arm Rouorw lnquwm Augwi 111-11. 2011,Mamie Bow Ra"Aan News, Aug. a/, ao haP://mynkWcbrtsumanmewa.mm/mMkbeazNufamanmex+ (lb.,nt- impaan,'sola-nrynle-benehums-ruraunm-i lanai ionsuuguorin Panv n.h[ml (vb, . W-oamb Mo. rtPoF-Fmn Myrdc a a om1 in rhe Aidck fo Piesaogduc haw nwY Wsoible Pains are avaihble.Por ala puepvea a(rhis Aad,C.•moon)nG faaan'sad oral 5 L(,aa,arc gnaliad ah, mgaho as nail vio6fi IFF H S,C.Cone Axa.Fib,6rv5 CM1.%N 5 L(vo�a)(v<,z&th.ii);Fd S J%aM S.C.Poh4c Hm4h,S.C.DV',OF HVt'm& Erm COxno4 hfgxJ/www.xAhv_gar/enwm�mena/mxTMNfow{hoN pmnmradngup (kn vbitod Co.1,won)(ve):S.,Ceps Axx.R[e5.6rv5 Ch.%NSE I.-noting Aaf'.iu n.,h rtvoked if a usaunm ha.a soon.hebw in fm&a-mnreawi.a routine mo ooloos); W. 5 K(ai);T.kphonc hi with faun.Buaaa of Road.Hol[h,S.G Dep[M Healal& Col.Conual Ulan.W.won)Ili);S.C.CUOeAxx.Aec5,6na5 Ch.%NS H(ak). 687 THE YALE LAW JOURNAL 122:574 2011 SITE SOURCE Se Wuie ST.Unm;Mo.,Cn't R[w[D Coe[ulaap 5 raoa.m(B)C. nulled ao.o)(u-dAain,, 'Regulamry Auabmi,');Kiel,&UvWvLnpa none 6f.as ali(ab);Td,phone ImeMew ankh Pat Mahoney,Sup aor,Farl&Beverage Comml,Pu,d 11,se Louis Dry,of Health(Mar,m, oaz) (a - "naming tin 'I'm art ao land inspmma, and rwt they lcciv5. w trod .mpeaiom);$ 1006,--. C"P"'tan C/LL4t,l S aa-my501)(D(as amaMN aoso) (ad).id 410-t-9(m,i[ g,d,aq ni,nit Telephone Imerview wish Pat Mahoony,supervisrc, Food&9eveng=C^nnol,FnM Dim.,Dep',of Heald U—p,von)in. ,ag,th);Dep',d Heald,OmV E'an'm mvm Hm4h RmiapnM Lupuwn Ra tarv,Cnv oe ST.Inv6,hapJ/sdou, -m,.gw/gov a m—depanmenae/L)(,);T ata,-w.h 86JfM-mnam);Np"a,nndnp �rt tan-J(bat viviad Ape 9,vou) Ci Tmup sup-ma t) lb.I ad(ra);Cep'e N Heahh, vv&Pat Ma ang,Cmam;4ymnn"pate.o- at sr.low lana,)Gb,a,dl:nits Omm,e Heiew xirh Par Mahonry,Supavimq FoW&Bevenga Gm[ml,Fall Div.,SL Units pep't or H<alah Qan.m,muj(ilj;sT.lOu6,MO.,CmRevLam Cm[0.gaaga(u amendd vo�o)(ak). Tnrnnm rnnmm; anayfr P,ug,.,,,.. ertv oe Towrrea, hapJ/ww"'.mmnno/IarWdiaaa(e (late isimd Peb,�s.zona)in,a);Fmf Pmava Aupntiaa and Dird S,,=ma:Fquamlf AaLe %-1 Cm of TOvnxao,hnpJ/apPmronrao/ l/FDFACJReg,lmj,(Lau vi lad Rb. o,a)lag,ah,ai, WssM1ingmn, Hall Reg.&Iinnnng Admin.,tml,6fery fggiene 6fvyarnn Smeeu Dianim(FSHISD),UC O.C, D.t.1 Hcuita,hnpJ/daM1.de.gav/rtrviae/o�aa(ery-M1ygiene-andinspadon-aervias-0ivisi,n Jim,viahed se,n'mta) Da, rt-,oung'de snH or de(food safary pm,,aml inaude p sannarians');Kiti,&DeW»1,-P-noa n,at p(,b,ul-noting ewb4shmena art mapeaad rt' me per Year);TdePhone I—x6t.with apmmlad-as Food S l Hyla—&Impaction Riva Div.(A,,6,aov) (,,-worming that inapeaom ivpea fwM atria arabhshmenu, mwmmmg pond,beauty patina,bade,shot',wl mlom,apt,elantni iie aubrhm,q and nage Pah);YdP wmhny m,DC,Yv ,hn V1—,dp—,W&(ma,my nam in'ra,-h fol boa;-Pea in aha anah mms;to rcsai¢to-Driving(,-i):.her4 da opuo,wda 'Disa,a'in ilit—aaoa)(rt-fining daa'ood aamnnt o,61a%ofwahes or'o,d; 1pa;'Gaury:'barb,:',ail vale,'maaag=:'poov,and'elamnipix in Wsshingm,,D.C., when 16 anah n&us is Junked m,mile);D-C M..pea.an.a,S aWaa.a(zo03)(d); Tdapho,e Inarvwvv oiah Aahlry BaWN,Fond pm[Div.,Washin,, n D.C.Dep',ofHealrh Oan. a3.a,a)Dd-darifyi,g de mk-based fmmm,,aryam;af.mrilyi,g 6.daea is no aa,damd pmu-g of..apeman—da); H.IIE Raguddon&Marring Admin.,SanA Imgwmm,D.C. D.1 nr Huiix,baW://wannmgmn.de.gegov.,nnJwebadmiNdld_nJweb/?a=msPmions(Ian nmed Apr.v,ata)DJ). 688 Yale Law Review: Conclusion(page 83 of pdf, page 656 of report): Targeted transparency remains one of the most promising regulatory approaches of this generation.As the rare instance in which disclosure can broadly affect behavior,it has the possibility to transform mandated disclosure into a genuine tool for empowering cognitively constrained consumers. This Article has shown, however,that even the perceived paragon of targeted transparency can be seriously flawed in implementation.our examination of over 700,000 inspections in San Diego,New York,and eight other jurisdictions shows that grades can be uninformative and costly.Targeted transparency cannot solve of avoid the core issues of administrative law—the institutional design of inspection agencies, the development of administrable rules and standards,and the accountability and oversight of expert agents.Without these elements in place, health inspections cannot generate meaningful information,and targeted transparency risks turning into a facile mantra of regulatory reform.To quote the"nudger in chief":"[D)isclosure may greatly alarm people. ..without giving them any useful information at all." Remarks of J.David Piposzar, MPH Wednesday,December 10,2014 Good evening. My name is David Piposzar,and I am here as a retired environmental health administrator with the Allegheny County Health Department having spent over 35 years in public health service.Following my retirement at the Health Department,I was appointed the first Executive Director of The Center for Public Health Preparedness at the University of Pittsburgh,Graduate School of Public Heatth,where I also taught public health as an Adjunct Assistant Professor in the Department of Environmental and Occupational Health. In the past,I served as a consultant to the Pennsylvania Department of Health on matters of public health preparedness,and assisted both the Conhmonwealth and our local health department in securing much needed funding for improved epidemiological support,improved laboratory capability,and rebuilding of core public health infrastructure. I hold a Bachelors degree in zoology,a Masters degree in public health. First,I want to emphasize that I am here this evening as a volunteer,and that 1 am not receiving any compensation whatsoever for any participation.In that regard,Ihold no allegiance to the health department, restaurant association,or political interest group.I am here because I am concerned about promoting sound public health policy: one that focuses first and foremost on preventing disease,promoting healthy behaviors, and protecting the public from known health hazards.My underlying concern is that we may be straying from our primary public health mission by not applying the facts and science as these relate to the efficacy of a restaurant grading system. In March of this year I took issue with the proposed restaurant grading system in a letter to the editor of the Pittsburgh Post Gazette in which I raised the issue of meaningless measures and the possibility of undo political interference in the delivery of public health in our County.I followed up with a literature review on restaurant grading systems in the United States,and formalized my findings in a letter addressed to each member of County Council dated March 13,2014.In that letter,I offered a set of criteria by which tojudge the merits of such a grading system In short,I asked them to consider six questions: 1)Will grading reduce the incidence of food-borne illness in the County? My research found little or no evidence to support claims that a grading system will reduce food-borne disease,patient visits for medical care,hospitalizations,or reported illnesses.Studies proclain ing such benefits did not take into account where illness originated,quite possibly originating in homes,institutions,schools,and establishments other than restaurants.Authors also admitted that in addition to grading,there was an increase in inspections, training of food handlers,and other factors that contributed to any observed reductions in disease. Furthermore, reports of Food-borne illness are vastly under reported and physicians may not accurately diagnose the cause of such outbreaks.Major recent outbreaks in the United States are more frequently associated with contaminated food supplies entering the market long before a family or restaumnl prepares a meal.Public health should be directing far more attention toward the entire continuum of food safety from"farm-to-table"by ensuring the safety and quality of food entering Allegheny County long before it reaches end-users like restaurants and home kitchens. 2)What impact will grading have on Health Department resources and the continued trust credibility and reputation of the Health Department?An important unintended consequence of grading systems is the shilling of much needed health department resources away from other more serious needs in order to defend challenges to the grade.The public might be better served if the health department utilized its limited resources in training and technical assistance, frontline disease surveillance,outbreak response,and public education. We also shouldn't miss an opportunity to develop industry"Partners for Food Safety"and to promote excellence in the industry above and beyond mere compliance with the food code.Furthermore,some studies have shown that grading places inspectors in a position of fudging the scoring, particularly around borderline facilities falling between A and B grades. In New York City the number of"A"ratings dramatically increased shortly after implementation resulting in unfounded claims of improved sanitation levels. Other studies have shown that sanitation levels remain largely unchanged.There is always the potential to`fix"the made "fudging the nudge"as one author concluded, which clearly undermines the trust and credibility of health department. My students always preferred an A-rather than a B+and argued vehemently even though test scores were the same. Grading is largely symbolic rather than substantive. 3)Will grading accurately convey the intended message to consumers? Grading systems used in other jurisdictions largely epitomize the term"window dressing".We shouldn't attach too much meaning to a single annual inspection,because it's not a fair reflection of what goes during the remaining 364 days of the year,nor does it take into account the past history of compliance over many years.It is simply a snapshot of conditions at that one point in time.It says nothing about whether it is safe to eat there.Consumers are ill served by a false sense of security attributed to an"A"grade. 4)Would some alternative method of reporting restaurant hygiene accomplish the same result?The health department has done a commendable job of making inspection reports available to the public either on- line or by issuing consumer alerts when restaurant conditions warrant.The green-yellow-red rating attached to these reports provides consumers an easily understood evaluation.If increased compliance with the food code is the goal of the food program,the health department has numerous avenues to address recalcitrant operators, including fines,penalties,closure and adverse publicity. An unsatisfactory grade warrants immediate action, not a grade posting. Another grading system may just bombard the consumer with too much information.(e.g. AAA,Michelin,Forbes,Zagat,Yelp, Good Food Guide)and it may actually contribute to more confusion within food industry.Why should Allegheny County adopt a system not used in the rest of the Pennsylvania? How many consumers understand what existing government ratings mean?(e.g. USDA grading of meat, poultry and dairy products). 5)Would inaction result in any harm?We have gotten along pretty well in the absence of a grading system so the medical practice adage,"First Do No Hann'might apply.It's not clear from my research whether a grading system will have much effect on consumer choices and some studies have shown that it doesn't seem to make much difference in restaurant hygiene. 6)My final criteria asked,"Is undo political pressure to enact regulations compromising the actual or perceived integrity,independence,and objectivity of the Board of Health and/or its Director?" Any possibility that the health department is viewed as a rubber stamp on issues of this importance needs to be addressed. We all should be cognizant that the Director and Board of health are appointees of the Chief Executive subject to Council's approval.I do know that the restaurant grading system was opposed by the former Director,Bruce Dixon and that an earlier A-B-C rating system was dropped years ago. When the makeup of the Board,the Directorship,and the administrative staff of the food program all change within a short period of time,there is reason to raise the possibility of undo influence in this matter. Let me be clear,I am supportive of posting hygiene scores and a pass/fail type rating system rather than a letter grade.I would expect inspectors to complete full and accurate inspections without bias or concern as to a restaurant's bottom line. Let consumers attach whatever value they see in those results. In my estimation a highly-charged value laden,letter grading system fails to recognize both the diversity within the food industry and the possibility of soliciting inspector favors for good grades. Whatever limited empirical evidence we have,suggests that grading does little to reduce disease or improve restaurant hygiene and on the other hand creates unnecessary anxiety,potential fudging of inspection results,and unwarranted consumer confidence in the safety of eating and drinking eslablislunents.We all should all be concerned as to why a new grading system is being advocated and be reassured that decisions of the.Health.Board and its Director are based on sound science and not unduly influenced by our elected officials or the media. Remarks of J. David Piposzar,MPH Wednesday,December 10,2014 Good evening. My name is David Piposzar,and I am here as a retired environmental health administrator with the Allegheny County Health Department having spent over 35 years in public health service. Following my retirement at the Health Department, I was appointed the first Executive Director of The Center for Public Health Preparedness at the University of Pittsburgh,Graduate School of Public Health where I al u l io taught b tc health as an Adjunct d3unc[Assistant Professor in the Department o ep f Environmental and Occupational p itonal Heahh. In the past,I served as a consultant to the Pennsylvania Department of Health on matters of P public health preparedness, and assisted both the Commonwealth and our local health department in securing much needed funding for improved epidemiological support,improved laboratory capability,and rebuilding of core public health infrastructure. I hold a Bachelors degree in zoology,a Masters degree in public health. Fast,I want to emphasize that I am here this evening as a volunteer,and that I am not receiving any compensation whatsoever for any participation.In that regard,I hold no allegiance to the health department, restaurant association,or political interest group.I am here because I am concerned about promoting sound public health policy;one that focuses first and foremost on preventing disease,promoting healthy behaviors, and protecting the public from known health hazards.My underlying concern is that we may be straying from our primary public health mission by not applying the facts and science as these relate to the efficacy of a restaurant grading system. In March of this year I took issue with the proposed restaurant grading system in a letter to the editor of the Pittsburgh Post Gazette in which 1 raised the issue of meaningless measures and the possibility of undo political interference in the delivery of public health in our County.I followed up with a literature review on restaurant grading systems in the United States,and formalized my findings in a letter addressed to each member of County Council dated March 13,2014.In that letter,I offered a set of criteria by which to judge the merits of such a grading system. ht short,I asked them to consider six questions: 1)Will grading reduce the incidence of food-borne illness in the County? My research found little or no evidence to support claims that a grading system will reduce food-borne disease,patient visits for medical care,hospitalizations,or reported illnesses. Studies proclaiming such benefits did not take into account where illness originated,quite possibly originating in homes,institutions,schools,and establishments other than restaurants.Authors also admitted that in addition to grading,there was an increase in inspections,training of food handlers,and other factors that contributed to any observed reductions in disease.Furthermore,reports of food-borne illness are vastly under reported and physicians may not accurately diagnose the cause of such outbreaks.Major recent outbreaks in the United States are more frequently associated with contaminated food supplies entering the market long before a family or restaurant prepares a meal.Public health should be directing far more attention toward the entire continuum of food safety from"farm-to-table"by ensuring the safety and quality of food entering Allegheny County long before it reaches end-users like restaurants and home kitchens. 2)What impact will grading have on Health Department resources and the continued trust credibility and reputation of the Health Department?An important unintended consequence of grading systems is the shifting of much needed health department resources away from other more serious needs in order to defend challenges to the grade.The public night be better served if the-hcalth department utilized its limited resources in training and technical assistance, frontline disease surveillance,outbreak response,and public education. We also shouldn't miss an opportunity to develop industry"Partners for Food Safety"and to promote excellence in the industry above and beyond mere compliance with the food code. Furthermore,some studies have shown that grading places inspectors in a position of fudging the scoring,particularly around borderline facilities falling between A and B grades.In New York City the number of"A"ratings dramatically increased shortly after unplementation resulting in unfounded claims of improved sanitation levels.Other studies have shown that sanitation levels remain largely unchanged.There is always the potential to"fix"the grade "fudging the nudge"as one author concluded, which clearly undernines the trust and credibility of a health department. My students always preferred an A-rather than a B+and argued vehemently even though test scores were the same. Grading is largely symbolic rather than substantive. 3)Will grading accurately convey the intended message to consumers? Grading systems used in other jurisdictions largely epitomize the term"window dressmg'.We shouldn't attach too much meaning to a single almost inspection,because it's not a fair reflection of what goes during the remaining 364 days of the year,nor does it take into account the past history of compliance over many years.It is simply a snapshot of conditions at that one point in tune. It says nothing about whether it is safe to cat there.Consumers are ill served by a Raise sense of security attributed to an"A"grade. 4)Would some alternative method of reporting restaurant hygiene accomplish the same result?The health department has done a commendable job of making inspection reports available to the public either on- line or by issuing consumer alerts when restaurant conditions warrant.The green-yellow-red rating attached to these reports provides consumers an easily understood evaluation.If increased compliance with the food code is the goal of the food program,the health department has numerous avenues to address recalcitrant operators, including fines,penalties,closure and adverse publicity.An unsatisfactory grade warrants immediate action, not a grade posting.Another grading system may just bombard the consumer with too much information.(e.g. AAA,Michelin,Forbes,Zagat, Yelp,Good Food Guide)and it may actually contribute to more confusion within food industry.Why should Allegheny County adopt a system not used in the rest of the Pennsylvania? How many consumers understand what existing government ratings mean?(e.g.USDA grading of meat, poultry and dairy products). 5)Would inaction result in any harm?We have gotten along pretty well in the absence of a grading system so the medical practice adage,"First Do No Hann"might apply.It's not clear from my research whether a grading system will have much effect on consumer choices and some studies have shown that it doesn't seem to make rrmch difference in restaurant hygiene. 6)My final criteria asked,"Is undo political pressure to enact regulations compromising the actual or perceived integrity,independence,and objectivity of the Board of Health and/or its Director?"Any possibility that the health department is viewed as a rubber stamp on issues of this importance needs to be addressed. We all should be cognizant that the Director and Board of health are appointees of the Chief Executive subject to Council's approval.I do know that the restaurant grading system was opposed by the former Director,Bruce Dixon and that an earlier A-B-C rating system was dropped years ago. When the makeup of the Board,the Directorship,and the administrative staff of the food program all change within a short period of time,there is reason to raise the possibility of undo influence in this matter. Let me be clear,I am supportive of posting hygiene scares and a pass/fail type rating system rather than a letter grade.I would expect inspectors to complete full and accurate inspections without bias or concern as to a restaurant's bottom line.Let consumers attach whatever value they see in those results.In my estimation a highly-charged value laden, letter grading system fails to recognize both the diversity within the food industry and the possibility of soliciting inspector favors for good grades.Whatever limited empiriW evidence we have,suggests that grading does little to reduce disease or improve restaurant hygiene and on the other hand creates unnecessary anxiety,potential fudging of inspection results,and unwarranted consumer confidence in the safety of eating and drinking establishments.We all should all be concerned as to why a new grading system is being advocated and be reassured that decisions of the.Health Board and its Director are based on sound science and not unduly influenced by our elected officials or the media. ONni ni n n ni n i niniunininn NO nni iu S i' N �u I II'rii°h N uiu1111 N 1111 ni m mn mi iuu i u u 1°i� °iiia in�m 0 iln ri�u IN i�ni imn !mi gni !nn ��mn jinn r t i � f a a L V J _ a N O 77 8 9 C C ° Z 1 O w a U Q f F 0 ,n5y a t 6 - 7 - o a - � o _ , T , g - , ) : w ] ) ] I § ; § f - ---- - - -- - --- - --- - - - \ \ \ , E \ \ \\ ! \ } ) \ } ) \ \ . � . X - -]� , (\ m ) Needham Public Health Division June 2019 Assist. Health Dir. -Tara Gurge Health Agents - Diana Acosta and Monica Pancare Activities Activity Notes Animal Permits 19-Animal Permits Issued. - All renewals were issued a new permit - 1 permit issued to Needham Golf Club(for 3 goats.) Beaver Permit 1—Issued 10-Day Emergency Beaver Permit to: - #292 Forest St.-Dave Volante at Volante Farms requested a permit due to flooding issues in his fields caused by beaver dams.Issued permit. Five beavers were removed. Bodyworks 1—Bodyworks Establishment Application Received - Baan Thai Spa 2—Bodyworks Practitioner Applications Received - Baan Thai Spa o Both applications pending Doctors note and CORI/SORI Results Demo Reviews/ 11-Demolition sign-offs: Approvals • 57 Hawthorne Ave • 116 Woodbine Circle • 77 Wayne Road • 304 Country Way • 21 Utica Road • 24 Bess Road • 53 Edwardel Road • 22 Parker Road • 290 Manning Street • 93 Dawson Drive 758 Webster St. Farmers Market 41—Farmers Market inspections conducted: - Initial inspections conducted by Diana and Stephanie McCulloch (intern). Follow up inspections have been conducted by Stephanie. 1—Farmers Market Application Received - Simple Sips Food—Temporary 10—Temporary Food Permits issued to: Food Event Permits - Charles River YMCA Independence Day 5k Oath Pizza @ Mitchell PMC Kids Ride Sweet Pete Ice Cream Boston Open Dancesport @ Powers Parent Talk @ Memorial Field-Flicks on the Field Sweet Tomatoes Mobile Truck Needham Community Council @ Rosemary Pool - Charity Lemonade Stand Firefly BBQ @ Brigham&Women's Physician BBD 5—Food Establishment looking to start their Pre-operation inspection process: Food—Food Permit - Eat Well—Updated plans need to be submitted since they have recently informed us of a Plan Reviews proposal to expand their seating area. Will submit a new plan for review. (Pending.) (Updates)/Permits - Servente Bakery&Cafe—Permit packet and fees received.Awaiting bathroom renovations. Issued Pre-operation insp. pending. Parche's Taoueria—Updated kitchen layout plan still pending for review.Additional plan Food—Food Permit review items received. Pre-operation inspection pending. Plan Reviews - Hungry Coyote—Submitted plan review packet. Pre-operation inspection pending. (Updates)/Permits Mike's Trattoria—Submitted plan review packet.Plan review items still pending. Scheduled Issued initial pre-operation inspection. 1—Food Establishment Plan Review Received Latina Kitchen&Bar—Still in plan review process. (Old L&K location.) 3—Food establishment seasonal/annual permit issued: - Forklift Catering—To take overvacant#301 Reservoir St. location. Tiffany's Brownies&Treats LLC—Residential kitchen. - W"nga[e at Needham Food Pre-operation 4—Pre-operation inspections conducted for: Inspections - NEW Sunv Williams School 1x11-Pre-operation inspections conducted. (Follow-up pending.) conducted - Forklift Catering 1x2)—Pre-operation inspection conducted.Permit issued. Tiffany's Brownies&Treats LLC full—Pre-operation inspection conducted. Permit issued. Food-Mobile 3—Mobile Truck Inspections conducted: - Chickenand Rice Guys—back up truck Sweet Tomatoes Mobile Truck - Sweet Pete's Ice Cream(temporary event) Food Complaints 1/1 —Food Complaints/Follow-ups: - Dragon Chef—Customer reported seeing staff working without gloves in the kitchen while preparing food. Diana followed up with the owners to notify and re teach staff that bare hand contact with ready to eat foods is not allowed. Housing— 1/1—Housing Complaints/Follow-ups Complaints/Follow- - Chambers Street(NHA)—Waiting to hear on status of unit.One inspection of the unit was UPS conducted but will need to conduct a re-inspection of this unit, prior to re-occupancy.(On- going.) Nuisance— 5/12—Nuisance Complaints/Follow-ups conducted for: Complaints/ - D mps[ers near us/Farmhouse/Hearth/Masala Art(1/41—UPDATE—Pest elimination Follow-ups progress being made. Weekly meetings continue to be conducted with Pest Control companies,along with establishment owners,to review progress.Daily dumpster checks are being alternatively conducted by Tara and Diana. EHS pest conducted a major dry ice treatment to eradicate population. Progress to continue to be monitored. (Shared dumpster enclosure to be built,along with rodent landscape exclusion of potential harborage areas, prior to the end of July.) - Gould Street(0/1)—Diana informed by radiation control that they tried reaching out to the resident with no answer. Diana was not able to get in contact with the resident and no calls have came in from the resident.They do not have a working voicemail to leave a message. (On going). - St Mary Street(011)—Order letter sent out.Tara and Diana to follow up with resident on July 81h(Done.)Will continue to conduct weekly inspections to check on progress. (On-going) - H pto Ave(1/11—Neighbor's trash has been overflowing again.Diana conducted a site visit and did not witness overflowing trash.The trash barrels on the front of the property were very full. Diana left a business card with a note to reach out.The resident has yet to call in to go over the issue. (On-going) - Oakcrest Road 10/1)—Resident with leaf pile called to report it is a compost pile that they actively turn over. - Canavan Circle(l/11—Resident's neighbor had various items in driveway and lawn was overgrown. In the past,the complai ,w it A/C units because animals chewed through.�cont n overgrowth. Diana conducted a site visit and left a business card with a note.The area had tall grasses and there were also chipmunks sited in the area.The neighbor contacted Diana and explained why the property was in that status as they had been out of state for a while and there was miscommunication with the landscapers and junk removal company.The neighbor was able to resolve the issues with the grass and items in the driveway.Discussed that chipmunks may have been the culprit in the chewed wires. - Charles River YMCA(1/2)-Member reported contracting an antibiotic resistant staph infection in February.Suspects the YMCA Pool as the members primary care doctor told them that it was possible to contract from swimming. Diana reached out and received the Nuisance— pool logs from YMCA for February.The chemical levels were in the appropriate ranges and Complaints/ the pool was not closed during this time period. Diana contacted the member again and Follow-ups (cont.) shared the info sheet from the Center for Disease Control on Pools and MRSA: https://www.cdc.goylhealthywater/odf/swimming/resources/mma-factsheet odf It is not likely that the infection came from the pool. - Sheraton Needham (1/1)—Guest called to report he had been bitten by bed bugs.The general manager of Sheraton is bed bug certified. Ecolab was immediately called out before the complaint was reported to the Public Health Division.The presence of bedbugs was confirmed by Ecolab.Ecolab initially treated the rooms above below and to the left of the affected room (the room to the right was occupied by longterm guests).Diana followed up with the hotel to confirm the mattress and box spring were removed from the room.The manager also reported the room to the right of the affected room would be inspected while the long term guests were not in the room. Registered Medical 2—Registered Medical Marijuana routine inspections conducted at: Marijuana Sira Naturals. Inc.-Needham Dispensary Dispensary/Annual - Sira Naturals. Inc.-Milford Grow site inspected by Tara Permit Renewal for 1—Registered Medical Marijuana Dispensary Permit issued to: Needham RMD - Sira Naturals. Inc. (in Needham) (NOTE:Sira Naturals,Inc.is still Interested in having a follow-up discussion with the BOH about the Vet/Senior discount option for their Needham dispensary.Will submit data for the board to review and attend an upcoming meeting.) Septic- 1—Septic Abandonment form submitted for: Abandonment Form - #260 Cartwright Rd. Septic -Soil Test 1—Soil Test/Perc Test conducted for: Application/Perc - 91 Wellesley Ave.—For septic system upgrade. (Plan submittal pending.) Tests conducted Septic—Enforceable 1-Enforceable Agreement Discussion held for: - Agreement #57 Walker Lane—Met with Town Engineering,Water and Sewer, and Conservation Discussion Depts.,along with homeowner,to discuss the timeline for the municipal sewer extension project to bring up the sewer line to his property. (PLEASE NOTE:DEP Enforceable Agreement set on this property which is due to expire of Dec.31,2019.)Homeowner will work on contacting additional sewer contractors to take on this proposed sewer extension job. Homeowner may ask for an extension from the BOH for this Order once he has exhausted all efforts to connect by the 2-year deadline.UPDATE—Discussion to be held at BOH meeting. Tobacco 10—Calls made to remind all tobacco establishments of mint menthol ban going into effect on July 1. Inspections in process to spot check that no flavored items are on shelves. Tobacco Hearing 1—Tobacco Compliance Check Violation Hearing scheduled with: - Hiren Patel—Owner of 7-Eleven Highland Ave.(ta BOH Meeting.)See summary in BOH packet. Trainings 1—MHOA Training - Mobile Food Vendor Program 1—AFDO Training - 123itl Annual Educational eren Waste Hauler 1-Waste Hauler Inspections conducted: Permits Issued - Republic 1—Waste Hauler Permits issued to: - Zaccaria Trucking Well—Approval to 1—Approval to Drill letter sent for an irrigation well to be drilled at: Drill - #403 Grove St.—By Mr.Thomas Ogden(REG If 552).Ogden Well&Pump Company,Inc. Planning Board Plan 0—Plan reviews conducted for Planning Board. Reviews Zoning Board of 3-Plan Reviews conducted for Zoning Board of Appeals/Comments sent for: Appeals Plan - 117 Rolling Lane—Comments send re:demolition notification requirements. Reviews - #324 Chestnut St.-Comments sent on the proposed food establishment at the site. New Indian catering/take out restaurant. (Vacant spot next to Sweet Tomatoes.) #150 Gould St.—Comments sent for proposed catering/commissary expansion.(Existing Baker's Best Catering.) Yearly ISO Biotech Biotech C0 0 0 1 0 0 0 0 0 0 0 I 1 2 2 registrations Bodywork Bodywork 0 0 0 0 0 4 2 8 0 0 0 0 14 14 6 11 Estab.Insp. Bodywork Bodywork 0 2 0 0 0 5 2 0 0 0 0 0 9 6 4 3 Estab.Permits Bodywork Bodywork 0 2 1 0 0 16 2 0 0 0 0 0 21 22 13 10 Pract. Permits Bottling Bottling 0 0 1 0 0 0 0 0 0 0 0 0 1 1 2 1Permitinsp. Dema 12 13 9 5 11 5 12 4 7 2 13 Il 104 105- 112 11 Demo reviews Animal Domestic 0 1 I 0 0 0 0 0 0 0 0 19 21 19 17 16 Animal permits/ Permits/Insp. 0 0 20 0 1 0 0 0 0 0 1 0 22 3 16 Inspections Food Service 16 8 23 25 16 13 20 9 22 15 15 18 200 225 198 209 Routine Insp. Food Service 3 1 0 1 0 1 1 0 0 1 1 3 12 32 37 35 Pre-aper.Insp. Retail 6 5 7 7 2 2 3 4 4 3 0 3 46 60 69 71 Routine insp. Resid.kitchen 0 1 0 0 1 0 0 1 2 0 1 0 6 8 7 11 Routine insp. Mobile 1 4 0 1 0 0 02_1 2 5 3 17 13 15 9 Routine Insp. Food Service 1 1 0 3 5 3 4 4 3 2 1 1 28 53 51 50 Re-insp. Food Annual/Seaso 2 1 0 0 1 129 1 0 0 1 2 3 140 171 177 176 nal Permits Service/retail 7 12 19 14 12 2 3 3 3 14 35 10 134 163 158 107 Temp.Food Food Service 9 1 10 0 0 0 0 0 0 1 16 0 37 29 62 54 permits/Inspe Farmers 1 0 1 0 0 0 0 0 1 5 6 0 14 14 7 9 Market Food Service permits Farmers 50 45 41 42 10 0 0 0 0 70 4 �n OrpeN ins P. Category lul Au S 0 N D 1 f M- A Mo lu fY '19 FY':18 FY FY' Notes/follow Food Service 0 2 0 2 1 1 1 31 1 41 21 1 18 20 13 21 New Compl/ 0 2 0 2 1 1 4 31 1 41 21 1 21 21 17 21 Follow-ups Food Service 0 1 1 3 4 3 2 11 2 11 11 1 20 42 33 32 Plan Reviews Food Service 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 Admin. Hearings Grease/SeptageGrease/Septage 0 0 0 0 0 21 0 0 0 0 0 0 21 24 24 29 Haulers Hauler Permits Housing(Chap II Annual routine 0 0 0 0 0 0 0 0 0 0 0 0 0 14 14 7 Housing) insp./ 0 0 0 0 0 0 0 0 0 0 0 0 0 5 4 4 Follow-up insp. Housing 2 4 11 3 4 1 3 0 2 0 1 1 22 22 7 18 New Compl./ 3 5 11 4 4 1 3 3 2 0 1 1 28 24 11 37 Follow-ups Hotel 0 0 01 0 0 31 0 0 0 0 0 0 3 3 3 3 Annual insp./ 0 0 01 0 -0 -0 1 D 0 0 0 0 0 0 0 0 Follow-ups Nuisance 2 5 51 4 4 4 1 4 6 8 71 5 55 42 30 44 NewCompl./ 2 5 51 4 4 4 1 4 8 8 121 12 69 42 45 50 Follow-ups Pools 1 4 0 01 11 7 0 0 0 0 7 0 201 12 13 9 Pool msp./ 1 5 01 01 01 1 1 0 0 0 4 0 12 7 83 Follow up Pools 11 2 11 01 01 7 1 0 0 0 7 0 19 12 9 9 Pool permits Pools 2 1 00 0 0 0 0 0 0 0 0 3 44 19 8 Pool plan reviews Pools 01 01 01 01 01 4 0 01 0 0 1 0 5 7 6 4 Pool variances Septic 0 0 0 1 2 1 1 1 0 1 1 1 9 S 18 8 Septic Abandon Addition to a Septic 1 0 0 0 0 1 0 0 0 0 0 0 2 2 5 9 home on septic plan rev/approval Septic 0 0 01 81 91 3 1 01 01 0 0 0 21 281 431 23 Install. lnsp. Septic 0 0 0 0 1 0 0 0 0 0 2 0 3 1 0 3 COC for repairs 28t„r„.Iz,'�tgFlo� 3 r FY' FY' FY' Notes/Follow Cote a Jul Au S O N D ;54 M A Ma lU FV'19 9 rl' trvt ti,+!„6 �ktt �c: . .r:':�. 18 17 16 UPCOC for Septic 0 0 0 0 2 1 0 0 0 0 4 4 3 3 complete septic system Septic 6 4 3 5 7 6 6 5 5 6 62 51 62 61 Info. requests Septic 0 C 0 0 0 0 0 1 0 0 1 2 6 8 Soil/PercTest. Septic 0 1 0 1 2 0 0 2 0 0 6 5 8 6 Const. permits Installer Septic 0 0 1 0 1 3 2 0 0 0 0 1 8 9 11 9 permits Septic 0 0 1 0 1 1 1 0 0 0 0 1 5 3 6 6 Installer Tests Septic 0 0 0 0 1 0 0 0 0 0 0 0 1 3 7 3 Deed Restrict. Septic 1 1 0 1 2 1 0 0 0 1 2 0 9 23 14 14 Plan reviews Disposal of Sharps 0 0 0 0 0 7 0 0 0 0 0 0 7 9 9 10 Sharps permits/Insp. permits/ 0 0 0 0 2 5 0 0 0 C 0 0 7 7 Inspections Plan review- Subdivision 0 C 0 0 0 0 0 0 0 1 0 0 1 0 3 31nsp.of lots/ 0 0 0 C C 0 0 0 0 0 1 0 1 0 1 0 Bond Releases Special Special Permit/Zoning 1 2 0 2 4 2 7 6 0 3 4 3 34 15 12 16 Permit/Zoning memos Tobacco Ec 0 0 0 0 0 30 0 0 0 0 0 0 10 11 12 13 permits o 4 1 3 2 0 0 1 1 2 0 0 0 14 18 25 25 Routine insp./ 1 0 0 0 0 0 0 0 2 0 0 0 3 3 6 7 Follow-up Compliance Tobacco 0 0 0 10 0 0 C 10 0 10 0 0 30 41 34 48 checks Tobacco 0 0 0 0 0 0 0 2 1 0 0 0 3 4 2 4_NZ.-C.Pl Compl.follow- 0 0 0 0 0 0 0 2 1 0 0 0 3 4 2 4 ups Trash Hauler Trash Haulers 0 0 0 0 0 0 0 0 3 10 3 1 17 14 26 30 permits Medical Waste Medical Waste 0 0 0 0 0 2 0 0 0 0 0 0 2 1 2 2 Hauler permits Haulers Permission to Wells 0 2 0 0 0 0 2 0 1 6 2 7 6 drill letters/ 0 0 0 0 0 0 0 1 0 Ow 1 0 3 O Well Permits 29FQe' FY 19 Priority FBI Risk Violations Chart (By Date) Restaurant I Ins . Date Priority violation Description 28-7-206.13(A)Tracking Powders, Pest Control 8 Monitoring- Comella's 7/16/2018 A tracking powder pesticide may not be used in a food establishment. Kitchen -Eliminate mouse infestation 9-3-301.11 (B)Preventing Contamination from Hands- Cookies By Except when washing fruits and vegetables,food Design 9/21/2018 employees may not contact exposed, Kitchen -Need gloves. ready-to-eat food with their bare hands and shall use suitable utensils such as deli tissue,spatulas, tons single-use gloves or dispensing equip. 28-7-204.11 Sanitizers- Chemical sanitizers, including chemical sanitizing Pollard solutions generated onsite,and other chemical Kitchen -Sanitizer in 3 bay sink was low,—100 Middle 9/24/2018 antimicrobials applied to food-contact surfaces ppm-should be 150-200 ppm. School shall meet the requirements specified in 40 CFR 180.940.Chemical sanitizers shall not exceed manufacture's label instructions. 33-3-501.15(A)Cooling Methods- Cooling shall be accomplished in accordance with the time and temperature criteria specified under 3-501-14 by using one or more of the following: Kitchen -Tortellini was at 68 F-Should be S 9/27/2018 placing food in shallow pans, separating the food Sebastian's into smaller or thinner portions; using rapid cold held in walk-in before being placed in the cooling equipment;stirring the food in a container salad bar. placed in an ice water bath;using containers that facilitate heat transfer;adding ice or other effective methods 4-501.110 Warewasher Wash Sol.Temp.- Ensure that low temp dishwash machine operates Gari 10/3/2018 at a minimum of 1207 wash 8 rinse.The Kitchen-110'F in wash cycle temperature of the wash solution in spray-type warewashers that use chemicals to sanitize may not be less than 120°F. 5402.13 Conveying Sewage- Sewage shall be conveyed to the point of disposal through an approved sanitary sewage Wingate at 11/6/2018 system or other system, including use of sewage Kitchen - Needham transport vehicles,waste retention tanks,pumps, Repair hand wash sink drain in dish room pipes, hoses,and connections that are constructed,maintained,and operated according to law. 4.703.11 Methods-Hol Water and Chemical- After being cleaned,equipment food-contact surfaces and utensils shall be sanitized in:hot Beth Israel water manual operations for at least 30 seconds, Deac, hot water mechanical achieving surface Kitchen- 12/15/2018 Hospital temperature ofg 160*Fstens as measured in an Wash Temperature 140-140F. Kitchen irreversible registering temperature r times r,or chemical(manual or mechanical)for times specified the EPA-registered label use instructions. 4-602.11 (A)Food-Contact Surfaces and Utensils-Equipment foodcontact surfaces and utensils shall be cleaned:before each use with a different type of raw animal food such as beef, Residences fish, Iamb,pork,or poultry;each time there is a Kitchen -Slicing machine blade has at Wingate 12/15/2018 change from working with raw foods to working encrusted food debris on rim of blade. with RTE foods;between uses with raw fruits and Thorough cleaning and sanitizing required. vegetables and with TCS lead;before using or storing a food temperature measuring device; and any time during the operation when e contamination may have occ Iteh[i-n lo�Honte.>f3 g ' Kitchen-A hot holding unit was found unplugged in the back of the kitchen on the 3-501.14(A)Cooling Cooked Foods-Cooling shelf which contained tomato sauce and Sweet cooked TCS foods shall be done within 2 hours another pan of meatballs. PIC slated she was Tomatoes 12/20!2018 from 135oF to 7o°F and then within 4 told to unplug it at 4 o'clock the meatballs and hours from 70°F to 41°F. sauce were in the danger zone products discarded tomato sauce was 106-F and meatballs were 110"F. Kitchen -Oven cleaner and other toxic Briarwood 7-201.11 Storage Separation-Poisonous or chemicals stored above prep sink and prep Healthcare 1/6Q019 toxic materials shall be stored so they cannot table area. contaminate food,equipment,utensils,linens. Store segregated and away from all food, Center and single-service and single use articles. e ui ment re areas. 3-302.11 (A)(2)Raw Animal Foods Separated Kitchen -Raw chicken stored above RTE from each other-Except when combined as foods. ingredients,separating types of mw animal foods This chicken was wrapped in a chef's coat. from each other such as beef,fish,lamb,pork Removed. Specific discussion w/PIC on and poultry during storage,preparation, holding, and display by:(a)Using separate equipment for Storage segregating product 5o i1 is more each type,or(b)Arranging each type of food in functional for him. Mandarin equipment so that cross contamination of one Cuisine 1119/2019 type with another is prevented and(c)preparing each type of food at different times or in separate areas. Kitchen— 3501.14(A)Cooling Cooked Foods- Large bulk pans cooked rice cooling Cooling cooked TCS foods shall be done within 2 improperly hours from 135°F to 70°F and then within 4 hours Rice was 129°F, 110°F. from 70°F to 41°F. -Kitchen-Bechamel WF Fresh fish filets 3.501.16(A)(2)(B)Proper Cold Holding 41 °F; Porcini mushrooms mix 52°F; Suggest Temps.-All cold TCS foods shall be held at 41°F prethilling pans in freezer to help in keeping or below. Eggs that have not been treated to food 41 F or below. Provide a lid or cover for destroy all viable Salmonellae shall be stored in this unit,and the other refrigerator roll top refrigerated equipment that maintains an ambient without the to in the farcomer-COS. Spiga 1/19/2019 it temperature of 45°F or less. Several coddled raw whole shell eggs were observed in a small pan insert in warn water. (This was used for making Carbonara sauce.) The inside is not fully cooked and in the danger zone. Suggest using pasteurized shell eggs. 3404.11 Foo tl Contact with Soiletl Items 3- 302.11 (A)(3)Using clean and sanitized Kitchen-Utensils stored Improperly after equipment-Food shall only contact surfaces of: cleaning and sanitizing above 3-Comp sink. Masala Art equipment and utensils that are cleaned and single-service and single-use articles; Restaurant 1/19/2019 sanitized; or linens,such as cloth napkins that are used to line a container for the service of foods AND are replaced each time the container is refilled for a new consumer. 3-501.16(A)(2)(B)Proper Coltl Holding Cafeteria-Fridge Is at 46°F.Service is to be Temps.-All cold TCS foods shall be held at 41°F done today. Forward receipt of service to or below. Eggs that have not been treated to Health Div.ASAP. End of service. destroy all viable Salmonellae shall be stored in refrigerated equipment that maintains an ambient Tomorrow's it temperature of 45°F or less. Lunch @ 1/29/2019 WCVB 3-501.16(A)(2)(B)Proper Cold Holding Kitchen-Chickpeas and tuna temps were Temps.-All cold TCS foods shall be held at high. PIC voluntarily discarded. 41°F or below.Eggs that have not been treated to destroy all viable Salmonellae shall be stored in refrigerated equipment that maintains an ambient air temperature of 45oF or less. 3-301.11 (B) Preventing Contamination Kitchen—Bare hand contact w/ready to eat from Hands -Except when washing fruits food. The James 2/9/2019 and vegetables,food employees may not Product discarded. Train and review with staff. contact exposed, ready-to-eat food with their bare hands and shall use suitable u such as deli tissue, s atula3,lon s df 111 tO.HOfIIC Pcu;C use gloves or dispensing equipment. 3-801.11 (C) Special Requirements Kitchen-Cook stated that they provide (Raw/Partially Cooked RTE) - omelet, sunny side undercooked eggs to adult -The following foods may not be served or day care participants. Cease and desist this offered for sale in a RTE form: raw animal practice, unless pasteurized shell eggs of foods such as raw fish, raw-marinated fish, pasteurized liquid eggs are used. raw molluscan shellfish, and steak tartare, a partially cooked animal food such as lightly cooked fish, rare meat, soft-cooked eggs Otrada that are made from raw eggs, and meringue; Adult Day 2/16/2019 and raw seed sprouts. Care 7-204.11 Sanitizers-Chemical sanitizers, Kitchen -Kitchen chlorine sanitizer in 3 comp including chemical sanitizing solutions sink was in excess of 200 ppm. Discussed w/ generated onsite,and other chemical chef DeMans and PIC. antimicrobials applied to food-contact surfaces shall meet the requirements specified in 40 CFR 180.940.Chemical sanitizers shall not exceed manufacture's label instructions. 3-304.11 Food Contact with Soiled Items- Kitchen-Micro greens,towel and other Food shall only contact surfaces of: products covered w/paper towels direct food equipment and utensils that are cleaned and contact.Discussion with Juan PIC sanitized;single-service and single-use articles; or linens,such as cloth napkins that are used to line a container for the service of Farmhouse foods AND are replaced each time the Restaurant 2/16/2019 container is refilled for a new consumer. 3-501.14(A)Cooling Cooked Foods- Kitchen-Bolognese Sauce 176°F in walk in Cooling cooked TCS foods shall be done cooler. COS. Discussion wl PIC on alternative within 2 hours from 135°F to 70°F and then methods to cool rapidly sauteed mushrooms within 4 hours from 70°F to 41 OF. on service line-cool quickly,keep below rim of food to assist in keeping well chilled. 3-701.11 Discarding or Reconditioning Store-Box of Cinnamon Toast Crunch was Unsafe, Adulterated, or Contaminated ripped into. Discarded by PIC. Food-A food that is unsafe, adulterated, or not honestly presented as specified shall be discarded or reconditioned. Food that is not 7-Eleven from an approved source shall be discarded. 36044A 2/212019 RTE food that may have been contaminated (Chestnut by an employee who has been restricted or Street) excluded shall be discarded. Food that is contaminated by food employees, consumers,or other persons through contact with their hands, bodily discharges, such as nasal or oral discharges,or other means shall be discarded. 3-501.16(A)(2)(B) Proper Cold Holding Store-'From Our Kitchenreach-in unit was Temps.-All cold TCS foods shall be held at observed at 46"F. May be on defrost- 41 OF efrost.41°F or below. Eggs that have not been treated to destroy all viable Salmonellae shall be stored in refrigerated equipment that Volante 222/2019 maintains an ambient air temperature of Farms 45°F or less. Basement- Employees were not able to report definite 3-501.15(A)Cooling Methods- time when soups were added into fridge. In ice -Cooling shall be accomplished in bath but still above 41'F. Recommend cooling accordance with the time and temp - criteria specified under 3-53914 b Eti[9'n t0 T10mL'] one or more of the following: placing food in shallow pans; separating the food into smaller or thinner portions; using rapid cooling equipment; stirring the food in a container placed in an ice water bath; using containers that facilitate heat transfer; adding ice or other effective methods. 3-304.11 Food Contact with Soiled Items- Kitchen -Raw TCS foods fish stored layered Food shall only contact surfacesof: paper towel. Paper towel is not food equipment and utensils that are cleaned and grade. Use approved food grade storage only. sanitized; single-service and single-use articles; or linens, such as cloth napkins that are used to line a container for the service of foods AND are replaced each time the container is refilled for a new consumer. New 3-302.11 (A)(2)Raw Animal Foods Kitchen - Garden 3/2/2019 Separated from each other- Except when TCS foods stored improperly in walk in Cooler. combined as ingredients, separating types of raw animal foods from each other such as Raw chicken stored above raw pork and large beef, fish, Iamb, pork and poultry during buckets of cabbage/vegetables were on lower storage, preparation, holding, and display shelf. Smaller prepared TCS animal foods by: (a) Using separate equipment for each were all stored on the same shelf in the walk- type,or(b)Arranging each type of food in in without segregation.Store all TCS foods equipment so that cross contamination of separated and segregated to prevent cross one type with another is prevented and(c) contamination-Provide proper and organized preparing each type of food at different times food storage plan within 3 days or less. or in separate areas. 3-202.11 (A)(C)(D)Temperature-TCS food shall be at a temperature of 41°F or below Kitchen -Two lobster tails on table at room when received. Raw eggs shall be received temp 60F. Product discarded in refrigerated equipment that maintains an ambient air temperature of 45°F or less. TCS food that is cooked to a proper temperature and received hot shall be at a temperature of 135°F or above. Sudbury Kitchen - Farms Chemic 3/15/2019 Methods-Hot Water and The 3 comp sinks in the meat room and prep Chemical-After being cleaned, equipment area in the basement were not set up properly. sanitized food contact surfaces and utensils shall for The middle rinse sink was used as a wash/ in: col water manual operations f at least seconds; hoe water mechanical soap sink without a clear clean water rinse achieving sudace temperature of 160°F as Prior to sanitizing. measured by an irreversible registering Sinks were not labeled properly-Wash/rinse/ temperature indicator; or chemical(manual sanitize. Provide additional training and proper or mechanical)for times specified the EPA- signage. registered label use instructions. 3-501.14(A) Cooling Cooked Foods- Cooling Kitchen-Bulk Sauces cooling improperly on Acapulcos 3/23/2019 cooked TCS foods shall bea 185F+counter. Cool properly. Enchilada sauce was done within 2 hours from 135°F to 70°F and then within 4 hours from 70°F to 41°F. PIC stated they were made 1.5 hours ago. 3-301.11 (B) Preventing Contamination from Hands-Except when washing fruits Cookies by and vegetables, food employees may not Kitchen -No gloves on site. Employees Design 3/27/2019 contact exposed, ready-to-eat food with their observed working with cookies without any bare hands and shall use suitable utensils gloves. Hand washing was noted. such as deli tissue, spatulas, tongs, single- use gloves or dispensing equipment. 3-304.11 Food Contact with Soiled Items - Walk in freezer-Frozen salmon stored inside Food shall only contact surfaces of. a previously used raw chicken box. Fuji 4/6/2019 equipment and These are considered single use only and Steakhouse utensils that are cleaned and sanitized; should not be used for food storage. single-service and single-use article - storage food grade containers linens, such as cloth napkins that are used to line a container for the service of foods AND are replaced each time the container is refilled for a new consumer. 3-304AII Food Contact with Soiled Items -Food shall only contact surfaces of: Kitchen-Side towels are used to cover food equipment and utensils that are cleaned and products and as absorbable cloths for eggrolls sanitized; single-service and single-use Use food grade materials only.Discussed w articles; or linens, such as cloth napkins that PIC recommendations. are used to line a container for the service of foods AND are replaced each time the container is refilled for a new consumer. 3-302.11 (A)(2) Raw Animal Foods Separated from each other- Except when combined as ingredients,separating types of raw animal foods from each other such as beef, fish, Iamb, pork and poultry during storage, Kitchen-Raw cutlets of Pork,salmon, pork preparation, holding,and display by: (a) and chicken were all stored in the same Using separate equipment for each type,or container on service line. Store all TCS foods (b)Arranging each type of food in equipment separated and segregated to prevent cross so that cross contamination of one type with contamination another is prevented and(c)preparing each type of food at different times or in separate areas. 3302.11 (A)(1)Raw Animal Foods Kitchen- Separated from RTE- Pr Raw meal stored above hummus. Food shall be protected from cross- Corrected on site- contamination by: (1) Separating raw animal foods during storage preparation, holding and display from: (a)Raw RTE food including other raw animal food such as fish Gyro and for sushi or molluscan shellfish or other raw Kebab 4/0/2019 RTE food such as fruits and vegetables, and House (b)cooked RTE food. Basement- Paint and varnish seen next to catering pans. 7-201.11 Storage Separation - Chemical cleaners stored near food and food -Poisonous or toxic materials shall be stored so they cannot contaminate food, equipment.All food must be stored together away from all chemicals. Chemicals should be equipment, utensils, linens, and single put on designated shelf. service and si ile use articles. 3304.11 Food Contact with Soiled Items - Food shall only contact Kitchen - surfaces of: equipment and utensils that are Raw pork was stored with raw chicken in walk cleaned and sanitized; single-service and in cooler.The raw chicken was exposed. single-use articles; or linens, such as cloth Product discarded. Store TCS food separated napkins that are used to line a container for and segregated. Train staff on proper food the service of foods AND are replaced each storage time the container is refilled for a new Gari consumer. Restaurant 4/15/2019 3-304.11 Food Contact with Soiled Items- Kitchen- Food shall only contact surfaces of: Paper towels used for hand drying were used equipment and utensils that are cleaned and to wrap Raw sushi fish,and other food items. sanitized; single-service and single-use Wrap and store with food grade products only articles; or linens, such as cloth napkins that such as pvc film,foil,food grade tissue and are used to line a container for the service of similar. Discussion with PIC foods AND are replaced each time the container is refilled for a new consumer. 34 /, . x'to Homc Prlge 4-602.11 (A) Food-Contact Surfaces and Utensils 4-602.11 (E)(1-3) Food-Contact Surfaces and Utensils-Equipment food-contact surfaces and utensils shall be cleaned: before each use with a different type of raw Olin animal food such as beef, fish, 1st Floor Kitchen- College 4/18/2019 Iamb, pork, or poultry;each time there is a Ice machine to clean out. change from working with raw foods to working with RTE foods; between uses with raw fruits and vegetables and with TCS food; before using or storing a food temperature measuring device; and any time during the operation when contamination may have occurred. Kitchen - Avita of 2301.11 Clean Condition - Cook observed with uncovered/unprotected Needham 4/20/2019 Food employees shall keep their hands and bum/cul on lower forearms. Gloves did not exposed portions of their arms clean. cover. Protect this area. 3-302.11 (A)(2) Raw Animal Foods Separated from each other Except when combined as ingredients, separating types of raw animal foods from each other such as beef,fish, Iamb, pork Kitchen-Raw chicken breasts in walk-in were and poultry during storage, preparation, Bakers 4/27/2019 holding, and display by: (a) Using separate stored above cooked chicken breasts,and Best equipment for each type, or(b)Arranging above rare beef cubes on each type of food in equipment so that cross sheet pans. COS contamination of one type with another is prevented and(c)preparing each type of food at different times or in separate areas. Kitchen-Hollandaise sauce made with raw eggs @127F.Was held at room temp on top 3-501.16(A)(1) Proper Hot Holding of service line. PIC stated that it was made at Temps.All hot TCS foods shall be held at 730am. Product discarded. Discussion with Fresco 5/4/2019 135°F or above except roasts cooked to a PIC to make batches more frequently to hold temperature and time as specified in 3 less than 4 hours,alternatives to use 403.11 (B). pasteurized eggs,or hot hold in water bath@135f or higher. Keep out of danger zone 40-135f 4-501.111 Manual Warewashing - Sanitization Water Temp 4-501.112 Mechanical Warewashing- Sanitization Water Temp- Kitchen - If immersion in hot water is used for It was observed that female cook did not sanitizing in a manual operation, the sanitize equipment properly in the 3 temperature the water shall be maintained compartment sink. She rinsed in in water from a[171°F or above, faucet, did not properly wash rinse or sanitize 3-601.14(A)Cooling Cooked Foods- to prevent contamination. The Rice -Cooling cooked TCS foods shall be done Kitchen- Barn 5/11/19 within 2 hours from 135°F to 70°F and then Basement-Observed 3 metal containers of within 4 hours from 70°F to 41°F. sauce/liquid cooling improperly 7-301.11 Separation - Poisonous or toxic materials Kitchen -Cleaning chemical spray bottle shall be stored so they cannot contaminate stored where cooked rice is located food, equipment, utensils, linens,and single service and single-use articles aseparating Butane torch stored on top of refrigerator next these materials by spacing or partitioning, to food equipment storage shelf and locating these materials in an area that is not above food, equipment, utensils. linens and sin le-service/ le us Retit�it to Honre.Pa�e MA 590.003(C) FC 3-301.11 (B)- Preventing Contamination from Hands 6- 301.11 Hand Cleanser Available- Except when washing fruits and vegetables,food employees may not contact Kitchen- Otrada 6/1/2019 exposed, RTE food with their bare hands Hand soap empty and shall use suitable utensils such as deli tissue, spatulas, tongs, single-use gloves, or dispensing equipment.Single use natural rubber latex gloves are not recommended for food contact in food establishments. 3-501.14(A) Cooling Cooked Foods - Kitchen -Rice wrapping and cooling Wingate at 6/8/19 Cooling cooked TCS foods shall be done improperly @121f; Meatballs wrapped and Needham within 2 hours from 135°F to 70°F and then cooling improperly 157f; Beef Gravy 143; within 4 hours from 70°F to 41° Discussion with PIC on proper cooling 8-103.12 Conformation with Approved Procedures/HACCP- Kitchen - -If the RA grants a variance or a HACCP Nonconformance with establish variance for plan is otherwise required,the permit holder sous vide cooking temperature logs incorrect shall comply with the HACCP plans and and missing data hot water bath was required procedures that are submitted and approved per Board of Health to be 160°but the hot as a basis for the modification or waiver water bath was in actually 145°proper internal temperatures of product was not charted. Datemarking of product in walk in non existent. Spiga 622/2019 3-501.15(A)Cooling Methods- Provide datemarking.discussion with PIC on Cooling shall be accomplished in adding additional columns of the existing chart accordance with the time and temperature to make it easily understood to be in criteria specified under 3-501-14 by using compliance. one or more of the following: placing food in shallow pans; separating the food into Kitchen -A porcini sauce was cooling smaller or thinner portions; using rapid improperly at room temp.Temp was 100F. cooling equipment; stirring the food in a Chef stated it was placed in container at container placed in an ice water bath; using approx 2.5 hours earlier. Product reheated to containers that facilitate heat transfer; 165f reconditioned and cooler in ice bath adding ice or other effective methods. 3-501.16(A)(2)(B) Proper Cold Holding Temps.-All cold TCS foods shall be held at Kitchen-Pizza prep unit in front was at 44°F. Sweet 627/2019 4 VF or below.Eggs that have not been treated to Maintenance to be performed later today. Tomatoes destroy all viable Salmonellae shall be stored in Send receipt refrigerated equipment that maintains an ambient air tem eratureof 45°For less. 4-703.11 Methods-Hot Water and Chemical 4-501.113 Warewasher- Kitchen -There was no sanitizer solution for Sanitization Pressure—The flow pressure the mechanical dishmachine.The bucket was of the fresh hot water sanitizing rinse in a empty, and Equipment and utensils that had warewashing machine, as measured in the been washed for not being sanitize properly water line immediately downstream or the dish machine operation was seized and in upstream from the fresh hot water sanitizing employee went to restaurant depot to Three rinse control value, shall be within the range purchase a new supply during the inspection Squares 6/29/2019 specified on the machine manufacturers equipment and utensils plates and flatware Restaurant data plate and may not be less than 5 psi or must be washed and sanitize properly to more than 30 psi. prevent contamination 3-501.14(B)Cooling TCS Ingredients at Kitchen -Egg wash for French Toast was Ambient Temperature -TCS foods shall be stored near the hot grill and was 76f.The pan cooled within 4 hours to 41°F or less if was stored in a double pan insert without ice. prepared from ingredients at ambient Only water remained. Product discarded temperatures, such as reconstituted foods and canned tuna. 3-302.11 (A)(1) Raw Animal Foods Kitchen -Raw chicken breast p ecata at sauce Separated from RTE -Food shall be prep area stored above RTE. Raw chicken Blue on protected from cross-contamination by (1) should be stored on lower shelf below all Highland 6282018 Separating raw animal foods during storage ready to eat foods. preparation, holding and di p ay froi Raw RTE food includingo raw Retto'!t to Hone Put ¢,i food such as fish for sushi or molluscan shellfish or other raw RTE food such as fruits and vegetables, and (b)cooked RTE food. 3-403.11 (A)(D) Reheating for Hot Holding Kitchen - Potentially hazardous foods that are being Sweet mashed potato on service line range reheated for hot holding shall be reheated so 90f-120f. Remove product and reheat and hold that all parts of the food reach a temperature at 135f or higher-COS. of at least 165°F for 15 seconds and shall be done rapidly—within two hours. 3-501.14(B)Cooling TCS Ingredients at Kitchen - Ambient Temperature-TCS foods shall be Buttermilk for dredging breaded product at cooled within 4 hours to 41°F or less if room temp 76F Product discarded. Maintain all prepared from ingredients at ambient TCS food at 41'F or below at all times temperatures, such as reconstituted foods and canned tuna. Kitchen- 3-501.16 (A)(1) Proper Hot Holding Whole muscle intact roast beef was in the alto Temps.-All hot TCS foods shall be held at sham oven hot holding at 130°F.The internal 135°F or above except roasts cooked to a temperature of the rare beef was 115°F temperature and time as specified in 3. registered by digital thermometer.The ambient 403.11 (B). temperature was 122°F as registered by an infrared thermometer. Hot hold at 135F or higher to reduce bacteria growth. 37 Retin'n to Home Page - � � ) } [ \ [ ( [ \ \ - \ � \ } \ �� { { - - : l : � - { , . \ , . } " � � � � . ! _ ! ; ; _ � ) ) \ � _ \ } - \ : - - � !| | � : � " { ; , �. , rl ! ! I� 3 } � . ! , ! J ! ! . ! . ) � � ) \ � � (§ ) (�| - - - - - - (� | , ; . _ � � ( / \ . \ §| |� | § }| |�| � ! !! .!� § |�| }i (( \! | i -) j i|| { (i |/! ` ^ \\\ � \ /} / � � � a} 2 \ [ � \ � ( 2 � . � _ � _ « � . ° - � - d < : d \ �7\� }§ � ` \\// \� }\ƒ\�� ^\ \\ 2\\ (�2� \ � � 17 00" Retail Food Pro ram Standards Enrollments as of 6/30/2019 l RSbb security Aristrict Agent, tl(IIY?own MOlher F%P imm:by, STATE Acli A en< A enc a Ainine, tdut y1ll Alabama 1 2 0 0 0 100 Alaska 1 1 0 0 0 Arizona 1 13 0 0 1 G^%.'' 97.,91 1 0 0 0 0 _^ 100 Cabfornia 1 34 0 3 4 '�:.: _ :89:86':' Colorado 1 31 1 counties) 1 1 98i06s Connecticut 2 0 ](2 cities,35 towns) 6 2 2879 1 0 0 0 0 a6o' 1 0 0 0 3 0 0 0 I Gcor is 2 84 2(5 counties) 0 0 1 3 0 0 0 1 0 ](44 counties) 0 1 Illinois 1 31 3n¢omiee.auuae,aaaaa) 1 1 Indiana 1 4 0 0 3 Iowa 1 10 2(14 counties) 1 0 1 0 0 0 0 Kentuclu 1 0 5(35 counties) 0 1 - -0 1 0 0 0 0 Maine 0 0 1 0 i Ma and 1 ] 0 1 0 Massachusetts 1 1 0 A 1 ,!;Os.86•�1f Mich an 1 13 ](25 counties) 1 0 AFLI .,t6&85' Minnesota 2 9 505 caunties) a 0tfl6: 1 0 0 0 0 Missouri 1 39 1(2 counties) 6 1 Montana 1 23 1t3 counties) 0 2 Nebraska 2 2 1(3 counties) 0 0 Nevada 1 2 0 1 0 New Hampshire 1 0 0 3 0 S New Jerse Y 1 d 1(iciN.]lowns) 5 0 Z�1 New Mexico 1 1 0 1 1 fir New York 1 d 0 1 North Carolina 1 42 2(6 counties) 0 0 2fr' _'North Dakota 1 3 2(12 counties) 0 0 �`=r (1 2 11 0 3 0 Oklahoma 1 2 0 0 Onation 2 8 0 Penn anis 1 2 0 3 0 B°45' i 0 0 0 0LaF'i::a100, 1 0 0 0 0 "e l 1 Q'.. :"a$Ulh Dakota" '" 0 0 0 1 0 :'Y'1 xEA$r9,0: Tennessee 2 5 0 0 0 Texas 1 25 4 14 counties,2 cities) 48 0 Utah 2 5 5(20 counties) 0 0 " 9L2Bs" ur was 2 5 101 counties,td rates) 5 0- m„r„. �,16SS25 Washin on 1 27 2(5 counties) 0 2 West Vi inia 1 2 1 1(9 counties.t city) 0 0 Waconsin 1 1 2] 1 2(4 cites.6 villages) 5 1 Wyoming 1 3 0 1 1 0iti`�"".'^38:91.: *:only state-agency enrolled; As:no state-agency enrolled. Level of835 Enrolled-agency:Stain-61',Terntory--5(American Samoa,Guem,Northern Mariam Islands,Pucno Riec VirginWants.)'.County- -485;District--]5(281 caucuses,26 cities,44 ti-ea ,li,dialles);Cil,-117,Town--66.Tribn-II Univ.-9,Fedrrol Agrnry-5;Valage--I. 99.74% (only one State without state-level agency enrolled)of the U.S.population that reside in a state in which one or more of the eligible state-level agencies with a retail food regulatory program has enrolled in the Voluntary National Retail Food Regulatory.Program Standards. 69.09% of the U. S.population that reside in a locality(city, county, parish,etc.) in which the eligible local-level agency with a retail food regulatory program has enrolled in the Voluntary National Retail Food Regulatory Program Standards*. * -populations(such as:Arkansas,Delaware,District of Columbia,Florida,Kansas,Louisiana, Mississippi, - Rhode Island,South Carolina, Vermont) in jurisdictions where no local-level retail food regulatory program exists will be included in this figure if the prevailing state-level agency has enrolled. FDA for d Retail Program APPLYStandards Grant The United States Food and Drug Administration (FDA) and the Association of Food and Drug Officials (AFDO) have partnered to provide funds for completion of projects and training that enhance conformance with the Voluntary National Retail Food Regulatory Program Standards (Retail Program Standards). Grant funding is open to state, local, tribal and territorial regulatory retail food programs that have enrolled in the Retail Program Standards. Award Rate for $ lo be Awarded for " e • -•• CalendarYear • r • • _ s 2M • _ Accepting • • • • - • • - October 1 19 Go to www.afdo.org/retailstandards Project Categories Small Projects (Category 1) Self-Assessments of all Standards,Verification Audits,or custom up to$3,000 projects that increase conformance with the Retail Program Standards. Moderate Projects(Category 2) For more ambitious projects (computer software systems,risk factor $10,000•$20,000 studies,development of a written compliance program,etc.). To assist jurisdictions in meeting the requirements of Standard 2: Training Projects(Category 3)- Trained Regulatory Staff/maintenance of FDA Standardization up to$3,000 (support for attendance at retail courses, FDA Regional,Seminars, and other training events). Food Protection Task Force To assist Jurisdictions in meeting the requirements of Standards Support Projects (Category 4) roodborne Illness and Food Defense Preparedness and Response up to $3,000 and/or Standard 7:industry and Community Relations - _ r Retail Food Program Standards NACCHO MENTORSHIP PROGRAM " iamlConnian ec`onfoA`"yHe1M1Health The Notional Connection for Local Public Health F N T z• About Our Program � t The National Association of County NACCHO's Mentorship Program ° and City Health Officials(NACCHO) supports retail food regulatory Mentorship Program encourages all programs in implementing these retail food regulatory programs to standards.The program provides use theVoluntary National Retail peer-to-peer assistance and Food Regulatory Program Stan- intensive technical supporttohelp r dards(Retail Program Standards), enforce the standards efficiently - ` lNentees receive up , The Food and Drug Administration and effectively.Experienced retail Co.58 '0I. (FDA) created these standards to food regulatory programs advise reinforce proper sanitation and participating agencies on how to reduce factors associated with the get started and best apply the Retail '�• ; t �a reyc 8 � �`_� occurrence of foodborne diseases. Program Standards to improve their By adopting these standards, retail food protection programs.Mentees .food regulatory programs can also gain insight on developing and prevent foodborne illnesses and implementing policies and protect the health of their procedures,completing self- communities. assessments,and conducting verification audits. Benefits of Mentorship Our mentorship program provides retail food regulatory programs with the opportunities to: • Share sample policies and procedures, resources, and tools with other retail food regulatory programs from across the country. • Learn from other retail food regulatory programs during conference calls and face-to-face meetings. Network with NACCHO, FDA and other retail food regulatory programs , ca �ptember�.tRc`gilestfortap�'17e 4 •e; s-� For more information visit: http://www.naccho.org/programs/ environmental-health/hazards/food-safety http://www.fda.gov/Food/GuidanceRegulation/ Retai I FoodProtection/Progra m5tandards/ Jennifer Li Amy Chang Senior Director, Program Environmental Health, Environmental Health Health &Disability AChang@nac,cho.org JLi@naccho.org 202-507-4221 202-507-4242 Voluntary National Retail Food Regulatory Program Standards - January 2017 j FDA Page I of 4 Voluntary National Retail Food Regulatory Program Standards - January 2017 "Standards of Excellence for Continual Improvement" Developed and recommended by the U.S.Food and Drug Administration with inputfrom federal,state,and local Regulatory officials,Industry,trade associations,academia,and consumers. OMB Control No.o9;o-o621 Expiration Date:o8-31-2o2o See additional PRA statement Paperwork Reduction Act of 1995(PDF-10OKB) (hupWumm,.reginfo-go o/public/reginfo/pm pdD Table of Contents Note:If you are having trouble opening a document please save the file to your computer first. To save a PDF file,right mouse click the link andselect Save Target As or Save Link As'Follow prompts to save the file to your desired location,taking care to remember where you saved the PDF hie. You may then open it within Acrobat Reader Overview of the Retail Program Standards • Introduction to the Standards(PDF-72KB)(/media/86636/download) • Program Standards Definitions(PDF-52KB)(/media/86651/download) Information about Enrolling and Participating in the Retail Program Standards • Clearinghouse Work Group Questions and Answers for implemented 2017 standards(PDF -773KB)(/media/1193o9/download)December 2018 • Clearinghouse Fact Sheet(PDF-557KB)(/media/128017/download)January 2017 - • Administrative Procedures(PDF-145KB)(/media/877oo/download) • FDA National Registry Report(FDA Form 3958)(PDF-1.1gMB) (/media/n n93/download) Standard 1: Regulatory Foundation • Standard 1(PDF-9oKB)(/media/86659/download) https://www.fda.gov/food/vo luntary-national-retai l-food-regu latory-program-standards/vol... 8/14/2019 Voluntary National Retail Food Regulatory Program Standards - January 2017 1 FDA Page 2 of 4 • Standard 1—Self-Assessment and Verification Audit Form(PDF-776KB) (/media/86676/download) • Standard 1—Self-Assessment Instructions and Worksheet for Part I (PDF-387KB) (/media/86683/download) • Standard 1—Self-Assessment Instmctions and Worksheet for Part II (PDF- LIMB) (/media/86942/download) • Standard 1 —Self-Assessment Instructions and Worksheet for Part III (PDF-234KB) (/media/86693/download) • Standard 1—Verification Audit Instructions and Worksheet for Part I(PDF-72oKB) (/media/86708/download) • Standard 1—Verification Audit Instructions and Worksheet for Part II(PDF- 687KB) (/media/86921/download) • Standard 1—Verification Audit Instructions and Worksheet for Part III(PDF-68�KB) (/media/86924/download) Standard 2:Trained Regulatory Staff - • Standard 2(PDF-149KB)(/media/86717/download) • Standard 2—Self-Assessment and Verification Audit Form(PDF-714KB) (/media/86723/download) • Standard 2—Self-Assessment Instructions and Worksheet(PDF-674KB) (/media/86737/download) • Standard 2—Verification Audit Instructions and Worksheet(PDF-71oKB) (/media/86744/download) • Standard 2—Appendix B-1: Curriculum for Retail Food Safety Inspection Officers(PDF- 37KB)(/media/86752/download) • Standard 2—Appendix B-2: CFP Field Training Manual (http://www.foodprotect.org/guides-documents/)&(http://www.fda.gov/about- fda/website-policies/website-disclaimer) (http://www.fda.gov/AboutFDA/AboutThis W ebsite/WebsitePolicies/Disclaimers/default.htm) • Standard 2 —Appendix B-3: Establishment Categories(PDF-1.13) (/media/86761/download) Standard 3: Inspection Program Based on HACCP Principles • Standard 3 (PDF-78KB) (/media/86769/download) • Standard 3—Self-Assessment and Verification Audit Form (PDF-334K13) (/media/86776/download) https://www.fda.gov/food/voluntary-national-retail-food-regulatory-program-standards/vol... 8/14/2019 Voluntary National Retail Food Regulatory Program Standards - January 2017 1 FDA Page 3 of 4 Standard 4: Uniform Inspection Program • Standard 4(PDF- 67KB) (/media/86785/download) • Standard 4-Self-Assessment and Verification Audit Form(PDF-765KB) (/media/86795/download) • Standard 4-Self-Assessment Instructions and Worksheet(PDF- IAMB) (/media/86807/download) Standard 5:Foodborne Illness and Food Defense Preparedness and Response • Standard 5(PDF-1o9KB)(/media/86813/download) • Standard 5-Self-Assessment and Verification Audit Form(PDF-972KB) (/media/86821/download) Standard 6: Compliance and Enforcement • Standard 6(PDF-86KB)(/media/86829/download) • Standard 6-Self-Assessment and Verification Audit Form (PDF-826KB) (/media/86836/download) • Standard 6-Self-Assessment Instructions and Worksheet(PDF- 1.1MB) (/media/86929/download) _ • Standard 6-Verification Audit Instructions and Worksheet(PDF-72oKB) (/media/92795/download) • Standard 6-Explanation of Statistical Model(PDF-87KB)(/media/86843/download) • Standard 6-Establishment File Summary(PDF- 614KB)(/media/92858/download) Standard 7:Industry and Community Relations • Standard 7(PDF-98KB) (/media/86853/download) • Standard 7-Self-Assessment and Verification Audit Form(PDF- 814KB) (/media/86935/download) • • Standard 7-Self-Assessment Instructions and Worksheet(PDF- 137KB) (/media/86861/download) Standard 8: Program Support and Resources • Standard 8(PDF- n1KB) (/media/86864/download) • Standard 8-Self-Assessment and Verification Audit Form(PDF- 883KB) (/media/86868/download) • Standard 8-Self-Assessment Instructions and Worksheet(PDF-7o6KB) (/media/86872/download) https://www.fda.gov/food/vo luntary-national-retai 1-food-regulatory-program-standards/vo 1... 8/14/2019 Voluntary National Retail Food Regulatory Program Standards - January 2017 1 FDA Page 4 of 4 • Standard 8 —Staffing Level(FTE to Inspection Ratio)Assessment Workbook Instruction Guide(http://www.foodprotect.org/guides-documents/) &(http://www.fda.gov/about- fda/website-policies/website-disclaimer) (http://www.fda.gov/AboutFDA/AboufFhis W ebsite/W ebsitePolicies/Disclaimers/default.htm) • Standard 8—Staffing Level(FTE to Inspection Ratio)Assessment Workbook (http://www.foodprotect.org/guides-documents/) Gr(http://www.fda.gov/about- fda/website-policies/website-disclaimer) (http://www.fda.gov/AboutFDA/AboutThis W ebsite/W ebsitePolicies/Disclaimers/default.htm) Standard 9: Program Assessment • Standard 9(PDF-92KB)(/media/86876/download) • Standard 9—Self-Assessment and Verification Audit Form(PDF-214KB) (/media/86888/download) Other Important Documents • Summary of Changes(PDF-65KB)(/media/87403/download) . https://www.fda.gov/food/voluntary-national-retail-food-regulatory-program-standards/vol... 8/14/2019 Page I of I WA ND NV co NE .y HI TX �i N 2 'S' ♦ q FRN MARIPNA GUAM AFRICAN 51 http://www.afdo.org/Resources/CustomPages/AFDO/GFY/MainMap.png 8/14/2019 Town of Needham Food Establishment Risk Factor Study July 12,2019 Tara Gurge,Assistant Director of Public Health Diana Acosta,Environmental Health Agent Pamela Ross-King, Project Consultant Stephanie Argetsinger,2018 Intern Michelle lovino, 2019 Intern Background The duty of public health is to prevent disease and promote the health and well-being of the community.One significant responsibility is to continuously monitor and improve food safety throughout public food establishments.There are over 250 identified foodborne diseases that can contribute to a variety of illnesses,from minor to severe symptoms. It is estimated that 48 million individuals get sick, 128,000 are hospitalized,and 3,000 die each year as a result.1 There are many ways food can be contaminated or compromised,therefore identifying direct causes and prevention methods is not any easy task. Many factors of potential risk must be evaluated and continuously enhanced in order to stay on top of food safety. The Town of Needham has a population of over 30,000 and 121 annual food permits.Needham joined the U.S. Food and Drug Administration's(FDA)Voluntary National Retail Food Regulatory Program Standards in 2017.The standards include nine areas of focus to achieve uniformity among state and local governments regarding food safety. Needham subsequently received a grant for assistance in completing Standard 9, Program Assessment, in order to analyze and reduce risk factors that contribute to foodborne illness.The five major risk factors include Poor Personal Hygiene,Contaminated Equipment, Improper Holding Temperatures, Improper Cooking Temperatures,and Unsafe Food Sources. Needham's risk factor survey was conducted by a consultant in 2018 in 85 randomly selected food establishments to determine town baseline conditions. Data was Men analyzed,and in 2019,focused interventions began.The ongoing interventions aim to reduce the risk of foodborne illness and increase food safety in both the establishments,and the risk factor areas,that were determined the most out of compliance i