Integrazione Stretta Di Robotica E Visione: Esempi Applicativi E Concetti Di Base

QuickTime™ and a decompressor are needed to see this picture.

Tight Integration of Machine Vision & Robotics with LabVIEW for automated assembly, packaging and testing Ignazio E M Piacentini [email protected] www.imaginglab.it

Giovedi 8 Ottobre 2009, Museo del Patrimonio Industriale, Bologna Martedi 13 Ottobre, Technotown – Villa Torlonia, Roma

QuickTime™ and a decompressor are needed to see this picture.

Agenda








Basic concepts and requirements Few application examples Standards in Machine Vision / lack of standards in robotics Direct communication from LabVIEW to the robots: introducing a robotics library / a robotics toolkit in LabVIEW Questions and answers

Tight integration of vision & robotics

2

QuickTime™ and a decompressor are needed to see this picture.

How and why did we
start integrating vision and robotics?





We have a strong background in machine vision, we saw the potential of adding vision to robots, we design integrated solutions ‘vision-first’ We strongly believe that there is an increasing number of applications suitable for fast, small payload robots We purchased our first robot (anthropomorphic) in 2004: 




We liked everything except the programming aspects!

We designed and developed our software library / toolkit in LabVIEW initially for our own internal use (Mitsubishi), we have done the same for the Denso robot (using bCap in strict collaboration with Denso Europe /Japan), work is progressing fast with Kuka, other brands will follow

Tight integration of vision & robotics

3

QuickTime™ and a decompressor are needed to see this picture.

Customers and applications Our customers are small / medium companies, often with strong mechanical background …or large companies / institutions
From neutrino detectors, to semiconductors, to packaging, to cosmetics, …
Automated assembly, testing, packaging, …
Some of the most interesting applications are covered by NDAs and cannot be shown in this presentation
We will see a limited number of video-clips
All projects are based on LabVIEW + IMAQ + ImagingLab robotics library
The robotics library for LabVIEW has been deployed on approx 50 different robotics / vision installations


Tight integration of vision & robotics

4

QuickTime™ and a decompressor are needed to see this picture.

The essence of vision-robotics integration: the equivalent of LabVIEW 3-icon demo






Acquire the image Extract the information Download info to robot & execute

Tight integration of vision & robotics

5

QuickTime™ and a decompressor are needed to see this picture.

Flexibility, configurability, usability







We can be very clever designer, use the best available technology, develop the best software code, but the ultimate verdict is with the end user / customer, or equally important with the operator of the automated machine Robotics and vision need to be considered as a whole single entity A clever hardware design must be complemented by an even smarter software design and implementation A good level of ‘end user autonomy’ must be designed into the vision-robotics system from the very beginning

Tight integration of vision & robotics

6

QuickTime™ and a decompressor are needed to see this picture.

Machine vision, GPM, and ….






















We will not enter into the details of machine vision but .. Geometric and conventional pattern matching need to be complemented by other checks Parts may have different orientations / views It is necessary to discriminate recognizable / pickable / conformal (QC) parts

Some inherent complexity can be removed with a ‘smart’ user interface The end user must be able to set pass / fail criteria, vision and robotics origin and zeros Auto-calibration tools are a must Tight integration of vision & robotics

7

QuickTime™ and a decompressor are needed to see this picture.

The vision platform: from panel PC to Smart Cameras


The hardware selection is dictated by the application needs: Conventional PC Industrial Panel PC PXI-based industrial PC (Win-based or RT OS)  Compact Vision System / Embedded Vision System (Real Time OS)  Smart Camera (Real Time OS)   





The software platform remains the same!! See Smart Cam videoclip

Tight integration of vision & robotics

8

QuickTime™ and a decompressor are needed to see this picture.

A quick intro to 3D imaging





3D imaging has a great potential in robotics applications Available technologies: laser scanning, stereo-imaging, time of flight ImagingLab is currently working on 3D and has recently released a LabVIEW wrapper for the Ranger series of SICK-IVP cameras Example: random bin-picking (as opposed to palletizing)

Tight integration of vision & robotics

9

QuickTime™ and a decompressor are needed to see this picture.

From 2D to 3D imaging: detecting flaws Detecting flaws on free-form objects are not very obviously detected.

Courtesy of AQSense

Maybe this one is clearly a flaw ...

But this one isn't so clear !!! Especially using “traditional” 2D vision 3rd December 2008

Tight integration of vision & robotics

QuickTime™ and a decompressor are needed to see this picture.

From 2D to 3D imaging: comparing with model If the scanned surfaces could be compared with their respective models, the “defects” would be made apparent.

Courtesy of AQSense

Above Tolerance

Zero Difference

Below Tolerance 3rd December 2008

Tight integration of vision & robotics

QuickTime™ and a decompressor are needed to see this picture.

From 2D to 3D imaging: alignment, calibration, …

Generally, scanned objects are not aligned with their respective models However, if a comparison must be done, both surfaces should be aligned, up to a specified accuracy.

Reference or2008 Model Point Cloud 3rd December

Scanned object with Model

Courtesy of AQSense

IMPORTANT ISSUEs: geometric calibrartion in 3D space, merge of multiple images, alignment with reference model

Tight integration of vision & robotics

Before Alignment

After Alignment

QuickTime™ and a decompressor are needed to see this picture.

A complex application: testing DIMM memory modules @ EEMS (ex Texas Instruments) (1’ 35’’ movie)







Fully automated functional testing of DIMM memory modules 64 interchangeable motherboards/ 256 memory slots 2 robots with common working space (collision avoidance) Full auto-guide / auto-learn / onboard camera Total testing cycle time compatible with production line Repetitive project

Tight integration of vision & robotics

13

QuickTime™ and a decompressor are needed to see this picture.

‘Smart’ pick and place for the cosmetic packaging














Fast picking from conveyor and accurate placement (< 0.1 mm) Many different sizes and shapes require flexible imaging configuration tool Most repetitive project using inverted Denso SCARA (22 robots, more to come ….) Triggered the development of the direct LabVIEW > b-Cap library Co-design with partner company Tight integration of vision & robotics

14

QuickTime™ and a decompressor are needed to see this picture.

‘Smart’ pick and place for the cosmetic packaging /2











Early testing of precision conveyor tracking (accurate on-the-fly deposition) Assembly of multicolour face-powder trousse Two robots / three vision systems Fast development of new machines

Tight integration of vision & robotics

15

QuickTime™ and a decompressor are needed to see this picture.

EST Projet (Educare per la Scienza e la Tecnologia): teaching vision & robotics at the ‘Leonardo da Vinci’ Science Museum in Milan Un progetto, anzi due, per il Museo della Scienza di Milano





Two ‘machines’ sponsored by NI, Mitsubishi, IL Recognising shape and colour, assembling and disassembling Interactive Taking care of an audience of variable age / designing for zero or little maintenance

Tight integration of vision & robotics

16

QuickTime™ and a decompressor are needed to see this picture.

Conveyor tracking in pharmaceutical packaging










Dual line with 2 vision systems / 4 DENSO SCARA + 2 FANUC Large selection of different items Curved transparent / translucid plastic Multiple pattern matching / multiple object views ……

Tight integration of vision & robotics

17

QuickTime™ and a decompressor are needed to see this picture.

Why integrators need to use different brands of robot






Standards exist for cameras, at least at communication / driver level: (old analogue standards at signal level, CCIR, RS170, etc.), IEEE 1394 Firewire, CameraLink, GigE, … Robots of different brands do not share a common programming language Integrators / end user are often faced with the need to ‘mix & match’ different robot brands Based on offer range and characteristics  Choice imposed by end customer 




Functionally all robots are similar (performances, reliability,etc. not withstanding)




An ‘abstraction layer’ can be built in LabVIEW Tight integration of vision & robotics

18

QuickTime™ and a decompressor are needed to see this picture.

Three different text-based programming examples Kuka code example Denso code example

1

INI

2

PTP HOME Vel= 100 % DEFAULT

3

GRIPPER_OPEN () ; Open the gripper

4

PTP P_PICK_UP Vel= 100 % P_PICK_UP Tool[1] Base[0] ; Move 100 mm above the pick position.

5

LIN_REL {Z -100} ; Move to the pick position.

6

GRIPPER_CLOSE () ; Close the gripper APPROACH P,P_PICK,100, S=100

7

LIN_REL {Z +100} ; Move 100 mm MOVE above the pick position. L, P_PICK, S=50

'!TITLE "Test Pick&Place"

Mitsubishi code example

PROGRAM TEST_PICKPLACE

10 SERVO ON

TAKEARM

20 OVRD 100

MOTOR ON

30 HOPEN 1

CALL GRIPPER OPEN

position.

' Open the gripper.

40 DLY 0.5 ' Open the gripper ' Move50 100 mm P_PICK,100 above the pick position. MOV 60 OVRD 50 ' Move to the pick

8

PTP P_PLACE_UP Vel= 100 % P_PLACE_UP Tool[1] Base[0] ; Move 100 mm aboveP_PICK the place position. 70 MVS

9

LIN_REL {Z -100} ; Move to the place position. DEPART P,100, S=100

CALL GRIPPER_CLOSE the pick position.

10

GRIPPER OPEN () ; Open the gripper

11

LIN_REL {Z +100} ; Move 100 mm above the place position.

12

PTP HOME Vel= 100 % DEFAULT

' Move 100 mm above the pick position.

' Close the gripper

80 HCLOSE 1 ' Move 100 mm above

' Move to the pick position. ' Close the gripper.

90 DLY 0.5

APPROACH P,P_PLACE,100, S=100 ' Move 100 mm above the place position. 100 OVRD 100 MOVE L, P_PLACE, S=50 position.

' Move to the place 110 MVS,100

' Move 100 mm above the pick position.

CALL GRIPPER_OPEN

120 MOV P_PLACE,100 ' Open the gripper

' Move 100 mm above the place position.

DEPART P,100, S=100 the place position.

130 OVRD 50' Move 100 mm above 140 MVS P_PLACE

' Move to the place position.

150 HOPEN 1

' Open the gripper.

MOTOR OFF GIVEARM 160 DLY 0.5 END 170 OVRD 100 180 SERVO OFF Tight integration of vision & robotics

19

QuickTime™ and a decompressor are needed to see this picture.

Three different communication / programming examples: Mitsubishi set-up


Client




Server

1;1;MOTOR ON

•String assembly •Motor-on icon •Generation of a ‘readable’ string •String sent to the controller (TCP-IP, serial, …) •The controller ‘parses’ the string and performs the command Tight integration of vision & robotics

20

QuickTime™ and a decompressor are needed to see this picture.

Three different communication / programming examples: Denso set-up


Client




Server

Data packet 001010011001010101001 010101010101010000111 101010100010101010101 01001

•Data packet assembly •Motor-on icon •Generation of data packet •Data packet sent to the controller (TCP-IP over Ethernet) •The controller ‘parses’ the data packet and performs the command Tight integration of vision & robotics

21

QuickTime™ and a decompressor are needed to see this picture.

Three different communication / programming examples: Kuka set-up


Client




Server

•VI Server technology (NI) •Motor-on icon •Instance of motor icon both at client and server level •‘Call’ to the LabVIEW program resident on the server •The controller performs the command Tight integration of vision & robotics

22

QuickTime™ and a decompressor are needed to see this picture.

A common LabVIEW ‘world’ for robotics, vision, feeders and HMI
A simplified diagram of a whole application Select vision recognition algorithm Enable MV quality check

Type of feeder

Type of robot

Pick & place coordinates

Select vision-based feeder management Tight integration of vision & robotics

23

QuickTime™ and a decompressor are needed to see this picture.

Building an ‘abstraction layer’: how to handle different brands of robots



An ‘action’ or command is essentially identical for different brands of robots A high level command (or program) common to ‘all’ robots is translated via a robot-specific plug-in

Command

High level LabVIEW robot protocol

Denso Protocol

Mitsubishi Protocol

Kuka Protocol

Staubli Protocol

…

XYZ Protocol

Robot Controller Tight integration of vision & robotics

24

QuickTime™ and a decompressor are needed to see this picture.

Official release and commercial availability of LAbVIEW robotics libraries ….







Denso Robotics Library has been released February 2009 by ImagingLab and will be by NI web services (ni.com): official announcement on the 25 November 2009 at IREX 2009, Tokio, and SPS-IPC, Nuremberg Mitsubishi Robotics Library is available now from ImagingLab Kuka Robotics Library will be released Q1-2010 (beta version available now on request) Other library, special needs, please ask ….. Tight integration of vision & robotics

25

QuickTime™ and a decompressor are needed to see this picture.

Questions & answers …

Tight integration of vision & robotics

26