Tomato application is a pillar category of facility agriculture in China. In the high-density planting environment of connected greenhouses, the fruit clusters are staggered, the branches and leaves are blocked, and the fruit clusters are shaken by air currents. The labor cost and intensity of manual harvesting are high.

For intelligent harvesting equipment, operational efficiency, positioning accuracy, and low fruit damage are interdependent and difficult to balance simultaneously, becoming the core technological bottleneck that restricts the large-scale implementation of equipment.

Organized by the Key Laboratory of Intelligent Harvesting Robots of the Ministry of Agriculture and Rural Affairs, Jicui Intelligent Manufacturing has formed a research and development team in collaboration with Jiangsu Academy of Agricultural Sciences and Nanjing Agricultural Mechanization Research Institute of the Ministry of Agriculture and Rural Affairs. The team avoids the design concept of simply stacking robotic arms and focuses on systematic technical research on multi arm harvesting configuration, visual recognition positioning, and integrated end effector for cutting and clamping. A 4SCM-50A facility tomato harvesting robot has been developed, which achieves high efficiency, high positioning accuracy, and low fruit damage harvesting operations through collaborative optimization of mechanism algorithm end effector software.

PART 01

Multi arm compound picking configuration

Collaborative scheduling achieves a significant increase in harvesting efficiency

The traditional single arm picking scheme has limited coverage and requires frequent adjustment of the machine's position to traverse the fruit clusters. The operation pace is slow and difficult to match the production efficiency requirements of large-scale greenhouses.

Simply increasing the number of robotic arms does not directly translate into improved work efficiency. In densely populated plant environments, problems such as arm movement interference, imbalanced task allocation, and ineffective waiting are prone to occur.

The research and development adopts a multi arm picking solution with a collaborative arm+dual-mode group three-axis linear sliding table composite configuration. The hardware is divided into a mobile sliding table module, a 6-axis collaborative robotic arm module, a visual perception module, and an end effector module. Each unit adopts standardized quick change interfaces for easy maintenance and quick replacement of end effectors.

The picture shows the STPE collaborative framework

On top of hardware, we innovatively build an STPE closed-loop integrated collaborative planning framework to solve the problem of multi arm parallel job conflicts at the algorithm level, and achieve safe and efficient collaborative scheduling in four major links:

one ️⃣ Spatial structuring: Cluster the picking targets and generate a set of gate point candidates, breaking down complex spatial picking tasks in a structured manner and reducing the computational cost of algorithm solving in high-density planting scenarios.

two ️⃣ Temporal collaboration: Cluster level gate point dual layer iterative solution strategy, completing task allocation and staggered scheduling for three robotic arms, avoiding arm space collisions from the source.

three ️⃣ Path planning: Based on the actual distribution of fruit clusters in the field, generate the optimal reference motion trajectory for each robotic arm.

four ️⃣ Execution correction: A multi rate closed-loop correction mechanism that combines high-frequency safety distance monitoring and low-frequency trajectory repair, dynamically adjusts the motion instructions of the robotic arm in response to on-site changes such as plant disturbance and fruit string shaking.

The picture shows the testing scenario of the picking robot

The simulation and prototype test results show that the average picking cycle time of the multi arm can reach 3.8 seconds per string, which is 7.3% -14.2% shorter than the traditional motion planning scheme, and the system robustness is significantly improved.

The picture shows the one click autonomous calibration software running display

The multi arm system requires strict consistency in the coordinate system of multiple machines, and coordinate deviations can directly cause the picking point to shift, leading to missed picking and fruit damage. To address the engineering pain point of low efficiency in industrial batch deployment calibration, a one click autonomous hand eye calibration software based on ArUco identification and depth point cloud matching has been developed to complete the coordinate mapping calculation between the camera and multiple robotic arms. The whole machine hand eye calibration time has been compressed from 1 hour to 15 minutes.

The picture shows the detection of measuring equipment such as laser trackers

At the same time, laser trackers are used to carry out orthogonal calibration of the coordinate system of the module robotic arm, correcting axial deviation and spatial coordinate errors, ensuring consistency in job positioning, and laying the foundation for efficient and stable picking.

PART 02

AI Visual Recognition and Localization System

Establish a precise foundation for harvesting operations

The greenhouse environment experiences severe fluctuations in lighting, with a large number of branches and leaves obstructing the area. The hanging vines and fruit clusters continue to sway with the wind, which can easily lead to missed targets and drift in picking point positioning.

The recognition and positioning accuracy of the visual system directly determines the precision of the picking operation, and is also a prerequisite for controlling fruit damage. The project builds a dual-mode visual system with "eyes on the hand+eyes outside the hand" to fully cover the entire workspace.

The picture shows a dual-mode visual system

The R&D team collected a large number of field samples, constructed a specialized dataset for greenhouse real working conditions, and developed a perception algorithm system suitable for complex agricultural scenes:

one ️ Lightweight maturity recognition algorithm: integrating LAMP channel pruning and SegNeXt attention mechanism to achieve model lightweighting, implementing high-speed inference on embedded hardware to determine the maturity status of fruit strings, and only picking mature fruits.

The picture shows the network organization diagram

two ️ Double picking key point detection algorithm: Combining the biological growth characteristics of tomatoes, output the two optimal cutting points of the fruit stem, providing precise execution targets for the end mechanism.

The figure shows the calculation process of double picking points

three ️ Slender fruit stem depth estimation algorithm: Based on ArUco true value calibration, depth noise filtering and coordinate mapping are completed to obtain the true three-dimensional spatial position of the fruit stem, solving the problem of long-distance ranging noise in depth cameras.

The image shows the accurate transformation from visual localization to robot execution coordinates

four ️ Dynamic trajectory prediction algorithm for shaking fruit strings: Adaptive Kalman filtering is used to recursively calculate the position and velocity of the shaking fruit strings, and a MS-OTG multi joint synchronous rolling update mechanism is constructed to continuously update the motion trajectory of the robotic arm during visual sampling gaps, achieving continuous tracking and positioning of the dynamic shaking fruit strings.

The figure shows a multi joint synchronous online trajectory generation method for shaking tomato picking

five ️ Actual measurement data of the whole machine: The average recognition accuracy of fruits is 97.8%, the positioning accuracy is 96%, and the algorithm inference speed is 22FPS. High precision perception can guide the end mechanism to accurately align with the fruit stem, reducing the risk of fruit damage caused by accidental touch, compression, and collision from the source.

PART 03

Cutting clip integrated end effector

The core execution unit for achieving low damage harvesting

The end effector is the core component for direct interaction between robots and crops, and is the final step in achieving low damage harvesting. To achieve reliable cutting of fruit stems while avoiding squeezing and bumping of fruits during clamping and cutting processes.

The project has independently developed an integrated harvesting hand claw with a cutting clip, which integrates a clamping mechanism and a cutting blade to achieve synchronous completion of clamping limit and fruit stem cutting. The institution adopts a lightweight modular structure made of aluminum alloy, with blades and clamping blocks that support quick disassembly and replacement. The clamping claw stroke is adjustable, suitable for different stem thicknesses and shapes of tomato clusters.

The picture shows the integrated picking gripper with scissors and clips

Under the synergistic effect of multi arm collaborative scheduling, high-precision visual positioning, and integrated end mechanism of shearing and clamping, the whole machine was tested in the field: the picking speed was 50 kg/h, the picking completion rate was 92%, the fruit damage rate was controlled at 3%, and the whole machine could operate continuously and stably for more than 4 hours, achieving the goals of high efficiency, high completion rate, and low damage in harvesting.

The picture shows the continuous operation test of the picking robot production scene

PART 04

Picking Assistant APP

A complete machine operation control software for greenhouse sites

The hardware structure and algorithm model need to be matched with human-computer interaction software to be implemented on the production site. Equipped with a self-developed picking assistant APP, it can be operated on tablets and mobile terminals, connecting the entire process of robot deployment, task issuance, status monitoring, and job data statistics, reducing the threshold for use.

The picture shows the interface of the picking assistant APP software

The main core technologies and functions of the software are divided into four modules:

one ️ Localized deployment and map management module: supports map loading, point marking, and picking area selection. Operators can select the picking operation area on the visual interface, issue picking tasks, and quickly deploy the robot operation scene.

one ️ Real time status monitoring module for the whole machine: real-time feedback of fault alarm information for the whole machine. When an abnormal situation occurs, the APP will immediately pop up an alarm, making it easy for on-site personnel to quickly handle it.

two ️ Visualization and Manual Intervention Module for Homework Process: Real time viewing of picking progress, supporting one click start, pause, and termination of tasks.

three ️ Homework data statistics module: automatically records core indicators such as single string picking time, picking success rate, and picking completion rate. Convenient for operation and maintenance personnel to review the results of their work, providing real field data support for algorithm parameter optimization and agronomic condition optimization.

The picture shows the human-computer interaction interface

The entire APP is deeply integrated with the underlying ROS control system of the robot, achieving a closed-loop system of perception, decision-making, execution, and human-machine control, allowing the picking system not only to have powerful ontology hardware and algorithm capabilities, but also to have operability for agricultural sites.

Innovation Convergence

Moving towards a new journey of smart agriculture

The difficulty of agricultural robots is not only the laboratory prototype effect, but also the engineering ability in complex and unstructured field environments.

This laboratory project will transfer the ontology design, multi axis motion control, and system integration engineering capabilities of industrial collaborative robots to the facility agriculture track, focusing on the complete chain of perception decision execution, and solving the common problems of "inaccurate recognition, unstable walking, difficult collaboration, and expensive landing" in the agricultural robot industry.

In the future, the laboratory will continue to delve into the field of agricultural robots, continuously iterate and optimize the series of facility tomato harvesting robot products, further improve the integration solution of agricultural machinery and agronomy, promote the wider implementation and promotion of intelligent harvesting equipment, assist in the transformation and upgrading of facility agriculture, and empower rural revitalization with machine, machine, and human technology